- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to use limited expert feedback to improve a frozen LLM agent. SynAct samples repeated rollouts and ranks decision points by action–outcome mutual information. A stronger model then turns selected examples into a short, reusable rule. SynAct applies this rule selectively when retrying failed attempts. The same signal also helps choose where an expert should step in at inference time. Experiments cover four benchmarks and four agent models. SynAct improves performance in all 16 training settings.
Critical strengths
-
A short expert-written rule improves the frozen agent without parameter updates or ongoing expert help. This makes the approach practical. After one iteration, the mean gain is 8.3 percentage points across 16 settings.
-
The baseline libraries in Appendix J use the same gate as SynAct but achieve about half of its gain. SynAct’s advantage under this control supports the value of its learned rules.
-
Appendix P keeps the routed tasks fixed and changes which decision the expert sees. Outcome-MI rescues 7 vs 5, 9 vs 6, and 8 vs 3 tasks across three models. These results support targeting expert feedback at a specific decision.
Critical weaknesses
-
Section 3.4 keeps successful first attempts and retries only failures. The main baselines in Table 5 add knowledge on every task. The paper should therefore limit its non-regression guarantee to deployment settings with a reliable success checker.
-
A retry under the same conditions, but without the learned rule, would help show what retrying alone adds. This matters because unaided performance varies by 3.4–8.5 percentage points across iterations.
Questions for authors
-
How do you check whether the first attempt succeeds at deployment? How do the gains change when methods use the same two-attempt setup? Please include a retry without a learned rule.
-
Appendix F summarizes the acquisition comparison under the final protocol. Can you show the full results and explain why the conclusion differs from Table 6? This would help show where outcome-MI adds value beyond the other steps.
3 comments
I appreciate the author's response and I think my concerns are resolved.
Thank you for the follow-up and for confirming that our response addresses your concerns. Given that these were the main concerns behind your initial assessment, would you consider reassessing your rating in light of the revision, including whether the revised paper warrants a clear accept? We would really appreciate it.
Thank you again for your careful review and for engaging with our response.
Gentle reminder: author-reviewer discussion period ending in a few days
Dear Reviewer iyeb,
Just a brief reminder that the author–reviewer discussion period concludes in a few days.
In our response, we ran the retry-only control you suggested in all 16 settings. Under an identical two-attempt protocol, the learned convention improves accuracy over a retry without it in 14 of 16 settings, by 4.3 points on average. We also clarified how first-attempt success is checked at deployment and conditioned the non-regression guarantee on that check. Finally, we isolated where outcome-MI adds value: with routed tasks, expert, and retry budget held fixed, anchoring the correction at the outcome-MI decision solves 146 of 566 tasks, compared with 126 for a random decision. We also replaced the earlier acquisition comparison with final-protocol results in all 16 settings.
If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.
Thank you again for the time and effort you put into the review.
Best regards,
Authors