- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper trains a notebook model to extract reusable knowledge from task trajectories while keeping the task model frozen. It combines teacher-based SFT with RL using rewards from subsequent tasks. Experiments on three benchmarks show improvements over SFT and transfer to longer task streams and unseen task models.
Critical strengths
Freezing the task model cleanly isolates the contribution of the notebook system. RL consistently improves over SFT across three benchmarks and two model sizes, and transfer to unseen task models makes the results more compelling.
Critical weaknesses
-
I am not sure how much the long-stream gains depend on continued knowledge accumulation. Figures 3--4 show cumulative success, which can keep increasing even if the notebook stops improving after the first few tasks. A comparison with a learned notebook frozen after eight updates would help establish whether continued updating is actually useful.
-
The evidence for the proposed credit assignment is less convincing than the overall benefit of RL. Figure 5 shows gains of 1.9 and 4.3 points over the outcome-reward baseline, but does not show uncertainty or variation across training runs. It is hard to tell whether these gains are robust enough to support the specific method.
Questions for authors
-
How does continued updating compare with freezing the notebook after eight updates on the same task streams? This would directly test whether updates beyond the training horizon remain useful.
-
Are the credit-assignment gains in Figure 5 consistent across training seeds? Could you report uncertainty for this comparison?
-
What exactly does the notebook model receive as post-task feedback on each benchmark? Does held-out evaluation provide only success/failure, or also detailed test diagnostics or reference solutions?
5 comments
Thank you again for taking the time to review our paper. As the discussion period is closing soon, we would greatly appreciate your feedback on our responses and the additional results. If you have any remaining concerns or would like further clarification, we would be happy to address them!
Thank you for the additional experiments and for clarifying the feedback available to the notebook model. It remains unclear how much of the improvement would carry over to settings where the model receives only success or failure feedback, so I am keeping my original rating.
We thank the reviewer for their time and for recognizing the consistent improvements from RL and the generalization to unseen task models. We conduct the requested freeze-after-8 control and a second training seed. We also discuss feedback with an example for each benchmark.
Thank you for the suggestion. On the same held-out ALFWorld streams, the trained notebook model updates the notebook for tasks 1-8, and the notebook is then either kept updating or frozen for tasks 9-40 (success %):
Continued updating improves success, mostly late in the stream. We will add this control for all benchmarks to the appendix, together with non-cumulative per-position success curves.
We trained an independent second run of both methods with different random seeds. The table reports the held-out AppWorld success rate after 160 training steps. The causal accumulated reward stays ahead of the trajectory-level outcome reward in both runs at 4B and 9B. We will add these results in the revision.
After a task is scored, the notebook model receives the task model's trajectory and a feedback string, which can affect only later tasks. Held-out evaluation provides the diagnostics of each benchmark's own evaluator, and never reference solutions. We provide a description and a short example of the feedback for each benchmark below:
Num Passed Tests: 1 Num Failed Tests: 6↵>> Failed Requirement: assert model changes match venmo.Notification, venmo.PaymentRequest↵AssertionError: {'venmo.Transaction', 'venmo.Notification', 'venmo.User'} == {'venmo.Notification', 'venmo.PaymentRequest'}ALFWorld binary outcome: FAIL. Executed 45 admissible actions and received cumulative native reward 0.↵Final environment feedback: You arrive at countertop 2. …↵Episode outcome: FAIL. The agent reached the 50-turn evaluation limit.GridGames episode terminated; reward=-1.000000; primitive_steps=1; invalid_actions=0↵Episode outcome: FAIL. The environment ended after 1 model turns.We would be very happy to continue the discussion and address any remaining concerns. If our responses have resolved your concerns, we would kindly ask you to consider reassessing your evaluation of our paper.