- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Anisotropic Program Anchoring (APA) to preserve behaviors not captured by the reward during RL fine-tuning. APA uses an executable symbolic program as a reference and applies dimension-specific regularization, allowing targeted dimensions to adapt while constraining others. Experiments on a multi-agent jazz accompaniment task demonstrate improved trade-offs between reward and harmonic competence, with the symbolic reference providing better competence preservation under distribution shift than a frozen behavioral clone.
Critical strengths
- The paper addresses an important problem: preserving behaviors not captured by the reward during RL fine-tuning.
- APA is simple and interpretable, combining dimension-specific regularization with an executable symbolic reference.
- The factorial experimental design separates the effects of regularization shape and reference choice, supported by multiple seeds and useful ablations.
- Results on jazz accompaniment demonstrate improved reward–competence trade-offs and better competence preservation under the tested distribution shift.
Critical weaknesses
- The methodological novelty appears limited, as APA mainly extends existing program-based anchoring with dimension-specific regularization strengths.
- Evaluation is restricted to a small, structured music task with predefined action dimensions and a reliable symbolic program. Generalization to less separable behaviors or imperfect programs remains untested.
Questions for authors
- How does APA compare with a simpler hybrid policy that uses the symbolic program for non-target dimensions and learns only the reward-targeted dimensions?
- How sensitive is APA to errors in the symbolic program? Could the authors provide a small experiment with an imperfect program to assess this dependence?
3 comments
Results for your Q1 (hybrid) and Q2 (imperfect programs) are now posted in our comment “Results of both additional experiments have arrived!” and in the revised PDF (Sec. 5.1 and Fig. 2a; Sec. 5.3; App. C.4; App. D.7). We would be glad to hear whether they address your questions.
Thanks for this clarification. I'll keep my original score.
Answer to Reviewer PQyJ
It's great to hear that you find the incomplete rewards problem important, APA simple, in a good way, and interpretable, our experimental design effective, and that you learned something interesting or useful. Thank you also for the fully positive feedback on paper soundness and our clear presentation.
We understand your skepticism. Overall, the contribution is more than the per-head weights, namely the separation of two choices the standard KL penalty fixes together. In distribution, the reference does not matter and the shape does (Tab.3); under shift, the shape and KL direction matters less and the reference does (Fig.3, the four clone conditions agree within 14%). This is now stated in the list of contributions (p.2).
Fair point. We just would like to add that the domain of symbolic music generation has a special dual tradition of rule-based program designs existing side-by-side with ML-based approaches (see also our discussion with Reviewer 15tt). We saw this big potential of combining both and aim to transfer what we learned from that tradition to other domains in future work, but decided to start with formalizing the methodology and testing it on a dense set of ablation experiments.
This is a particularly valuable remark, thank you. We have implemented experiments on that setting, which are now running. We will report their results as soon as they arrive. They will supplement Fig.2(a). Result and design will be discussed in a separate comment, as Reviewer xiQX also asked for exactly that.
The out-of-distribution setting (Sec.5.2) already represents a imperfect reference. However, we fully agree that a dedicated experiment is of particular interest here. So we used the recent days to implement an experiment using an imperfect version of our Jazz trio. They also touch the problem of less seperable behavior as the best-rewarded behavior would need to change the top note's octave for the major reward, but also the PC for the full reward. We will explain the experiment design and provide the results as soon as they arrive in the separate comment.