acceptodds
Under review as a conference paper at ICLR 2027

SynAct: Synergistic Active Feedback Learning for LLM Agents

Abstract

LLM agents executing multi-step workflows fail in two fundamentally different ways: recoverable slips, which further attempts can repair, and knowledge gaps, which retrying cannot resolve. Expert feedback can address the latter, but expert attention is scarce. The key challenge is therefore not whether to ask for feedback, but where feedback can change the outcome. Existing approaches either avoid expert feedback, allowing agents to reinforce their own errors, or allocate feedback at the task level without identifying consequential decisions. We present SynAct, an active learning framework that treats expert attention as the binding constraint and allocates feedback at the level of individual decisions. SynAct ranks decisions by the mutual information between the selected action and the terminal outcome across repeated rollouts, queries an expert at the highest-value decisions, and distills the resulting corrections into transferable rules that the agent can reuse without expert access at deployment. We theoretically characterize what this signal identifies under finite repeated sampling: with rollouts and at least observations per action, outcome-constant decisions have zero mutual information, while outcome-separating decisions have mutual information at least bits, yielding exact separation for any threshold . This also explains why success-based acquisition can be blind when all rollouts fail: success is constant even when failed runs reach different outcomes. Across four agent benchmarks and four models spanning three families, SynAct improves held-out accuracy in all 16 evaluated settings, by pp on average. The same signal transfers to inference-time routing, where SynAct outperforms budget-matched cascade, uncertainty, and expert-judge routers in 15 of 16 settings, while a single targeted correction outperforms seven additional unaimed retries.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, active learning, expert feedback, mutual information, feedback allocation, experiential learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iyebOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to use limited expert feedback to improve a frozen LLM agent. SynAct samples repeated rollouts and ranks decision points by action–outcome mutual information. A stronger model then turns selected examples into a short, reusable rule. SynAct applies this rule selectively when retrying failed attempts. The same signal also helps choose where an expert should step in at inference time. Experiments cover four benchmarks and four agent models. SynAct improves performance in all 16 training settings.

Critical strengths
  1. A short expert-written rule improves the frozen agent without parameter updates or ongoing expert help. This makes the approach practical. After one iteration, the mean gain is 8.3 percentage points across 16 settings.

  2. The baseline libraries in Appendix J use the same gate as SynAct but achieve about half of its gain. SynAct’s advantage under this control supports the value of its learned rules.

  3. Appendix P keeps the routed tasks fixed and changes which decision the expert sees. Outcome-MI rescues 7 vs 5, 9 vs 6, and 8 vs 3 tasks across three models. These results support targeting expert feedback at a specific decision.

Critical weaknesses
  1. Section 3.4 keeps successful first attempts and retries only failures. The main baselines in Table 5 add knowledge on every task. The paper should therefore limit its non-regression guarantee to deployment settings with a reliable success checker.

  2. A retry under the same conditions, but without the learned rule, would help show what retrying alone adds. This matters because unaided performance varies by 3.4–8.5 percentage points across iterations.

Questions for authors
  1. How do you check whether the first attempt succeeds at deployment? How do the gains change when methods use the same two-attempt setup? Please include a retry without a learned rule.

  2. Appendix F summarizes the acquisition comparison under the final protocol. Can you show the full results and explain why the conclusion differs from Table 6? This would help show where outcome-MI adds value beyond the other steps.

3 comments
AuthorsOpenReview ↗

Gentle reminder: author-reviewer discussion period ending in a few days

Dear Reviewer iyeb,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we ran the retry-only control you suggested in all 16 settings. Under an identical two-attempt protocol, the learned convention improves accuracy over a retry without it in 14 of 16 settings, by 4.3 points on average. We also clarified how first-attempt success is checked at deployment and conditioned the non-regression guarantee on that check. Finally, we isolated where outcome-MI adds value: with routed tasks, expert, and retry budget held fixed, anchoring the correction at the outcome-MI decision solves 146 of 566 tasks, compared with 126 for a random decision. We also replaced the earlier acquisition comparison with final-protocol results in all 16 settings.

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

I appreciate the author's response and I think my concerns are resolved.

Thank you for the follow-up and for confirming that our response addresses your concerns. Given that these were the main concerns behind your initial assessment, would you consider reassessing your rating in light of the revision, including whether the revised paper warrants a clear accept? We would really appreciate it.

Thank you again for your careful review and for engaging with our response.

Reviewer_outhOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents SynAct, which allocates scarse expert feedback to consequential decision points in LLM agent trajectories via mutual information compute from repeated rollouts, distills corrections into gated conventions, and improve accuracy across 16 evaluated settings while also supporting inference time routing.

Critical strengths
  • Principled, label-free acquisition of consequential decision points via I(A;O) that remains informative on all-fail tasks.

  • Consistent gains across 16 setting plus a clean non-regression guarantee from selective gating.

Critical weaknesses
  • Heavy dependence on a strong LLM expert as a proxy for human expertise, human authoring cost and quality remain untested.

  • Limited evaluation of robustness when state keys are imperfect or environment are stochastic.

Questions for authors

What is the expected cost and quality of the authored conventions when a human domain expert replaces Claude Opus? Do the same decision points remain high-value, and do the resulting rules transfer similarly?

1 comment
AuthorsOpenReview ↗

Gentle reminder: author-reviewer discussion period ending in a few days

Dear Reviewer outh,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we clarified that acquisition does not depend on the expert, so a human expert would review the same selected decisions, and we quantified the material the expert reads. We also showed that expert quality matters, since replacing Claude Opus 4.6 with Claude Sonnet 4.6 lowers the gain, and the revision now states the lack of a human-expert study as an explicit limitation. On robustness, conventions learned in the deterministic setting keep their gains under a stochastic τ-bench user (11.9 and 13.0 points for Qwen3-32B and Ministral-3-3B). The acquired decisions also remain largely unchanged when we randomly merge state keys.

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

Reviewer_yZf3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper develops a system for identifying errors in multi-step problem solving, and obtaining expert feedback on how to resolve those errors. They do this by running the model multiple times, identifying points where multiple decisions are taken, measuring the mutual information between the decision actions and the final outcome, and obtaining expert feedback (from a frontier model) on that decision to provide clarifying instructions. This expert feedback is then used at test time to adjust the prompt to help the model select better actions.

They consider a run with k=8 trajectories. If all actions give the same outcome, then their empirical mutual information is zero. On the other hand, if the actions are perfectly distinguish the outcomes (there is no overlap in outcomes), then the minimum entropy is about 0.8 (which occurs when the frequency of actions is most lopsided (2 vs 6). [I'm not exactly sure why this upper end is the right range to evaluate.]

In their experiments they run 8 trajectories across several data sets. Against baselines, they find between 5% and 10% inprovement over no-expert baselines. But their MI method was indistinguishable from other methods for selecting where to get expert advice.

Detailed Comments:

  1. I wonder whether shannon mutual information is the best mutual information for the setting of the paper. Specifically, Shannon MI has no unbiased estimator with finite samples, while other MIs do. See Kong (2026).

Kong, Y. (2026). The Algebraic Limits of Polynomial Information Measures. arXiv preprint arXiv:2606.14225.

  1. I am very nervious about the scalability of how they use the expert advice. The expert advice is used to add to the prompt. This addition is made after an error is discovered. I assume this is for scalability reasons, but it means that we need to be able to detect errors. On the other hand, if the expert advice is added to every prompt without detecting errors first, then it doesn't scale. That limits applications to settings were errors are easily detectable. I would have found it more interesting to post-train the base model with this expert advice. Since the method does not need verify outcomes to select where to get advice, it would be nice if it didn't need to verify outcomes when being used.

  2. I also wonder, if we are only running in settings where we can verify the success or failure, then why do we not only care about success/failure outcomes? They attempted to address this by saying that e.g., if all actions fail, but give different outcomes then their method finds this siguation. On the other hand, isn't it a better sign to address this action that all actions failed?

Critical strengths
  • develops a method for selecting where to get expert advice without ground truth.

  • it integrates mutual information in a novel way.

Critical weaknesses
  • either requires verifiable outcomes to use or doesn't scale.

  • its use of biased estimates of mutual information is a little ad hoc.

Questions for authors

I'd be happy for my detailed comments (1)-(3) to be answered as questions.

3 comments
AuthorsOpenReview ↗

Gentle reminder: author-reviewer discussion period ending in a few days

Dear Reviewer yZf3,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we tested whether the finite-sample bias of plug-in MI changes the selected decisions. With Miller–Madow correction, the selected decision is unchanged in all six tested settings (rank correlations of 0.98–1.00), and plug-in MI is more stable than the exactly unbiased U-statistic alternative you suggested in 13 of 14 settings. We also clarified which components use success information: acquisition is label-free, while gated deployment requires a task-specific success check, and Section 3.4 now conditions the non-regression guarantee on it. Finally, we explained why outcomes are needed beyond success labels (outcome-MI still ranks decisions when all rollouts fail, as in 20 of 49 acquired Qwen3-32B TextCraft tasks) and why the separation threshold is H2(m/k)H_2(m/k).

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

I appreciate the authors' discussion of other MIs. For me, this was helpful. I might not be the typical reader, and so the authors should feel free not to include the additional discussion in the paper if it distracts from the main message.

I do not have further questions about the paper at this time.

Thank you for the follow-up. We're glad that the discussion of alternative MI estimators was helpful.

Given that you have no further questions, would you consider reassessing your rating based on the clarifications and results in our response? We would really appreciate it.

Thank you again for your careful review and for engaging with our response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.