- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce a new way to automatically generate a reward program, SOR, with the goal of helping the policy repeat actions that should be repeated multiple times, like gathering wood. This method builds the reward program based on the achievements and actions the previous agent took. They test their procedure on Crafter and Craftax, showing an improvement over the PPO baseline.
Critical strengths
How to design a better reward function when the reward for actions that are stepping stones for later problems is a real problem. A model might only perform an action a couple of times, and if it doesn’t get any additional reward signal, it might stop, although it would be useful to repeat the action for other rewards. The idea of giving an agent an additional reward for steps that should be repeated multiple times is interesting.
Critical weaknesses
The method increases computational cost. So it could be that PPO, which spends the number of seen environment steps on training instead of training the policy that then redefines the reward, performs better.
While there is some degree of automation, the method still relies on human information. Between the similar games Crafter and Craftax, part of the method had to be adjusted. This reduces the benefit of automation. Additionally, the method can only adjust relative to what the previous policy found, bounding its progress by that of the policy used to make the reward program.
Questions for authors
Did you compare against PPO in a step-matched setting? Producing the setup for the environment seems to involve similar work effort to hand-drafting a reward without performing better. What advantage do we get from automation when considerable human effort has to be spent setting up the reward program without outperforming the hand-drafted baseline?
5 comments
Response to Reviewer ogCP (2/2)
Regarding weakness 2 and question 1: human input, automation, and reference-policy limitations
SOR automates the construction of the concrete reward program. Appendix C specifies the division between human configuration and automatic processing:
| Stage | Human specification or verification | Automatic processing and output |
|---|---|---|
| Data interface | State fields, action encoding, native first-unlock labels, episode boundaries | Collect trajectories and generate fitting labels from native logs |
| Event detection | Action/difference grammar and fitting/acceptance criteria | Search candidates, select by BIC, perform five-fold validation, output detectors |
| Recurrence structure | Counting, partitioning, support, and level-retention conventions | Replay trajectories, count recurrences, select groups and levels |
| Reward configuration | Reference budget, global reward parameters, verification and freezing | Compute support, group budgets, and per-level payments; export the program |
The human settings specify what to observe, how to count, which criteria to apply, and the total reward budget. They apply across candidates. Which events survive, how many recurrence levels receive rewards, and how much budget each group receives are determined by the data and construction rules. Interfaces and counting conventions connect rewards to actual environment operations; acceptance criteria control detector reliability; global parameters control reward scale and reference cost.
For the final programs in the main experiments, the procedure generates all 12/12 detectors, 114/114 recurrence levels, and 12/12 group budgets in Crafter, and all 7/7 detectors, 86/86 levels, and 7/7 group budgets in Craftax. Thus, 0% of these individual program entries are hand-specified; human work concerns the interfaces, global design, parameter selection, and correctness checks above. The new single-reference programs are reconstructed through the same procedure (Appendix K.1).
The manual programs in the paper are strong references developed by the authors using task knowledge and multiple experimental iterations (Section 2, Appendix E). SOR replaces item-by-item reward design with a procedure that can be rerun on new reference trajectories. In the added reference-quality experiments, changing the reference budget while keeping the construction procedure fixed generates new rules, levels, and budgets. The 500k reference produces results close to the 1M reference without manually rewriting individual rewards (Section 4.3, Figure 4; Appendix K.3).
Reference coverage determines which events the auxiliary program can reward, while the downstream policy continues learning from the complete native reward (Section 3.1). For example, the Craftax auxiliary program does not directly reward crafting stone pickaxes or stone swords, yet their success rates increase from 2.0% to 44.4% and from 2.5% to 49.6%, respectively (Section 4.4, Figure 5). Intermediate capabilities reinforced by the auxiliary program can therefore support progress beyond its own event list.
We sincerely hope that our response could address your concerns. Thank you once again for your valuable efforts and continued support!
Best regards,
All authors
Thank you for running the additional experiments. I agree that the amortisation of costs is a potential argument for SOR. Are there any possibilities to show that this method amortises costs better than, say, using the PPO policy? For example, a PPO policy run for the same number of steps as the first phase of SOR could be used to distill an AD policy. Without amortisation, native return doesn’t seem to be better than plain PPO on Crafter (slightly better at 5M steps and slightly worse at the other 3 checkpoints, all within noise) and slightly worse on Craftax. I understand that there isn’t much time for an additional experiment, so if an additional run showing the advantage on amortisation isn’t possible, it would be good to note the missing comparisons in the limitations.
Response to Reviewer ogCP (1/2)
Dear Reviewer ogCP,
Thank you for recognizing the motivation for recurrence feedback. According to your comments, we have added experiments and clarifications addressing total interaction costs, automation, and reference coverage.
Regarding weakness 1 and question 1: matching total environment interactions
The original experiments primarily fixed downstream training budgets and did not sufficiently examine the total cost of generating references. We have now added comparisons that account for reference-policy training, trajectory collection, and downstream training; full results are in Appendices K.3 and K.5. At the same total budget of approximately 5M Crafter interactions, SOR uses 21.4% fewer downstream training steps than PPO (3.932M versus 5.001M), while achieving a slightly higher mean native return: 13.07 ± 0.42 versus 12.88 ± 0.32 (five runs, mean ± SD; native return averages the latest 50 completed episodes).
The reference stage produces a reusable, frozen reward program. The downstream policy learns from scratch without inheriting the reference policy's parameters. We therefore evaluate both the program's benefit to downstream learning and the total cost of constructing and using it from scratch. T-REX, Section 5.2 and VIPER, Sections 4.1–4.2 also distinguish reference-data generation, reward construction, and downstream training. In SOR, the constructed program can be reused: our AD experiments use the program built from PPO references without training new reference policies for AD.
We also added reference-cost experiments, reducing two references to one and reference training to 500k steps. With 1M downstream steps, the 500k reference yields Score/native return of 19.32/11.48, close to 19.59/11.60 with a single 1M reference. Compared with the original two-reference configuration, reference training and collection decrease by approximately 73%, and total interactions including downstream training decrease by 50%. These experiments show that reference count and training budget can be adjusted to reduce preparation costs while retaining useful feedback (Section 4.3, Figure 4; Appendices K.1 and K.3).