acceptodds
Under review as a conference paper at ICLR 2027

Reward Moves What It Measures: Anisotropic Program Anchoring for RL Fine-Tuning under Incomplete Objectives

Abstract

Reinforcement learning fine-tuning optimizes generative policies against objectives that are incomplete: a reward scores what it can measure, never everything the designer wants, and optimization then degrades the unmeasured behavior. The standard remedy is a scalar KL penalty toward a frozen clone of the initial policy. This remedy has two limitations: it holds every behavioral dimension equally, so it can only slow the whole policy, and its reference is a snapshot that loses validity once the policy leaves the states it was fitted on. We propose Anisotropic Program Anchoring (APA), which anchors the policy to an executable symbolic program instead. The program encodes the unmeasured behavior directly and can be queried at any state the policy reaches, and one anchor coefficient per behavioral dimension lets the reward move what it measures while the program holds the rest. We evaluate APA on a musical task where the objective is incomplete by nature: a piano and a bass agent fine-tuned together to accompany a jazz soloist. In a factorial design, we compare ten experimental conditions on held-out measures of harmonic competence that the reward never scores. Without anchoring, competence falls by half. APA keeps competence within 1% of its initial value while realizing 80% of the unanchored reward gain, against about 55% for scalar anchors. Under distribution shift it loses 4% of competence where an anchor to a frozen clone loses 38%. Adding another reward term for the most visible failure repairs only that property, while others keep degrading.

Primary area:
reinforcement learning
Keywords:
reinforcement learning fine-tuning, incomplete objectives, reward hacking, anisotropic regularization, symbolic priors, neuro-symbolic reinforcement learning, multi-agent reinforcement learning, behavior cloning, music generation, algorithmic composition, jazz improvisation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PQyJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Anisotropic Program Anchoring (APA) to preserve behaviors not captured by the reward during RL fine-tuning. APA uses an executable symbolic program as a reference and applies dimension-specific regularization, allowing targeted dimensions to adapt while constraining others. Experiments on a multi-agent jazz accompaniment task demonstrate improved trade-offs between reward and harmonic competence, with the symbolic reference providing better competence preservation under distribution shift than a frozen behavioral clone.

Critical strengths
  1. The paper addresses an important problem: preserving behaviors not captured by the reward during RL fine-tuning.
  2. APA is simple and interpretable, combining dimension-specific regularization with an executable symbolic reference.
  3. The factorial experimental design separates the effects of regularization shape and reference choice, supported by multiple seeds and useful ablations.
  4. Results on jazz accompaniment demonstrate improved reward–competence trade-offs and better competence preservation under the tested distribution shift.
Critical weaknesses
  1. The methodological novelty appears limited, as APA mainly extends existing program-based anchoring with dimension-specific regularization strengths.
  2. Evaluation is restricted to a small, structured music task with predefined action dimensions and a reliable symbolic program. Generalization to less separable behaviors or imperfect programs remains untested.
Questions for authors
  1. How does APA compare with a simpler hybrid policy that uses the symbolic program for non-target dimensions and learns only the reward-targeted dimensions?
  2. How sensitive is APA to errors in the symbolic program? Could the authors provide a small experiment with an imperfect program to assess this dependence?
3 comments
AuthorsOpenReview ↗

Answer to Reviewer PQyJ

It's great to hear that you find the incomplete rewards problem important, APA simple, in a good way, and interpretable, our experimental design effective, and that you learned something interesting or useful. Thank you also for the fully positive feedback on paper soundness and our clear presentation.

Critical Weaknesses:

  1. The methodological novelty appears limited, as APA mainly extends existing program-based anchoring with dimension-specific regularization strengths.

We understand your skepticism. Overall, the contribution is more than the per-head weights, namely the separation of two choices the standard KL penalty fixes together. In distribution, the reference does not matter and the shape does (Tab.3); under shift, the shape and KL direction matters less and the reference does (Fig.3, the four clone conditions agree within 14%). This is now stated in the list of contributions (p.2).

  1. Evaluation is restricted to a small, structured music task with predefined action dimensions and a reliable symbolic program. Generalization to less separable behaviors or imperfect programs remains untested.

Fair point. We just would like to add that the domain of symbolic music generation has a special dual tradition of rule-based program designs existing side-by-side with ML-based approaches (see also our discussion with Reviewer 15tt). We saw this big potential of combining both and aim to transfer what we learned from that tradition to other domains in future work, but decided to start with formalizing the methodology and testing it on a dense set of ablation experiments.

Questions For Authors:

  1. How does APA compare with a simpler hybrid policy that uses the symbolic program for non-target dimensions and learns only the reward-targeted dimensions?

This is a particularly valuable remark, thank you. We have implemented experiments on that setting, which are now running. We will report their results as soon as they arrive. They will supplement Fig.2(a). Result and design will be discussed in a separate comment, as Reviewer xiQX also asked for exactly that.

  1. How sensitive is APA to errors in the symbolic program? Could the authors provide a small experiment with an imperfect program to assess this dependence?

The out-of-distribution setting (Sec.5.2) already represents a imperfect reference. However, we fully agree that a dedicated experiment is of particular interest here. So we used the recent days to implement an experiment using an imperfect version of our Jazz trio. They also touch the problem of less seperable behavior as the best-rewarded behavior would need to change the top note's octave for the major reward, but also the PC for the full reward. We will explain the experiment design and provide the results as soon as they arrive in the separate comment.

Results for your Q1 (hybrid) and Q2 (imperfect programs) are now posted in our comment “Results of both additional experiments have arrived!” and in the revised PDF (Sec. 5.1 and Fig. 2a; Sec. 5.3; App. C.4; App. D.7). We would be glad to hear whether they address your questions.

Thanks for this clarification. I'll keep my original score.

Reviewer_15ttOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces an RL algorithm, Anisotropic Program Anchoring (APA), which anchors a model during RL fine-tuning against a symbolic program with anisotropic weights for KL penalties for different action dimensions. The paper identifies that reward functions can fail to measure important criteria for success and the lack of such specification can cause a model to get worse on the unmeasured behavior. The standard KL penalty, according to the paper, applies the same KL constraint on all behaviors which prevents the model from being able to optimize some behaviors more than others.

Critical strengths
  1. The paper tests the proposed method on an interesting testbed for music generation. The particular experiment setting is multi-agent and dynamic which is reasonably difficult, and the reward function is constructed in a manner that deliberately fails to measure behaviors along some dimensions, thus making the testbed an appropriate one for the problem being discussed in the paper.

  2. The paper conducts a relatively dense set of ablations to understand the strengths and weaknesses of some existing approaches. For example, the paper attempts to reduce learning rate and clip thresholds to see if it slows down the degradation of performance and also considers random initialization versus behavior cloning initialization.

  3. Section 5.2 shows a particularly interesting result where anisotropic anchoring is shown to be the main driver of performance, since replacing the symbolic program with a BC anchor leads to similar results and then discusses how this is true only when the state distribution of the actor is not very different from the base model. This is a valuable result for an understanding of the importance of state distribution being similar. I do think, however, this insight should be cited since it was observed in prior work as well, such as in TRPO.

Critical weaknesses
  1. It is unclear why one might be able to easily construct a rule-based generator for tasks in general. For example, during a standard RLVR/RLHF pipeline, we fine-tune language models on tasks ranging from creative writing to question answering as a helpful assistant to mathematical reasoning, and for any of these tasks, constructing such rule-based generators may either be impossible or, at the very least, extremely expensive.

  2. Continuing from point (1), on such more general tasks, it is unclear whether the effort spent on constructing such rule-based generators is better spent on directly addressing the reward specification problem by constructing more grounded reward functions (for e.g., refining rubrics heavily for language model judges or allowing adaptive rubrics to address the possibility that a model gets good along some dimensions while not improving/degrading along others).

  3. In addition, when such a rule-based symbolic program is available, I am not sure why naive DAgger could not work just as well as APA and a direct comparison against that would help the paper.

  4. It is unclear whether, for general tasks, one can factorize the action space into independent components the way the paper factorizes, say, the piano’s top note into pitch class, octave and volume. In particular, without this factorization, I am not sure that the anisotropic weights for KL penalties can even be instantiated. Insofar as the paper’s two main contributions are replacing the base model with a symbolic program generator and anisotropic anchoring, and since both of them seem impractical in general settings, I am not convinced of the practical usefulness of APA as an algorithm.

  5. Furthermore, in the event that such factorization can be done, the existing approaches in the literature for better reward specification could potentially also work almost just as well.

It is possible that the paper’s focus is on music generation tasks only and that can plausibly justify the assumption that accessing such a symbolic program is realistic. However, if that is the case, the paper should clarify so and not claim that it is addressing the general problem of reward misspecification in RL fine-tuning. The current framing causes the reader to expect that the algorithm is being proposed as an alternative in general RL frameworks.

Questions for authors

Please see the section of critical weaknesses.

3 comments
AuthorsOpenReview ↗

Answer to Reviewer 15tt (part 1)

Thanks a lot for your valuable input. We were happy to read that you found our testbed interesting, the experiment setting reasonably difficult, the ablations dense, and that you highlighted our finding from Sec.5.2 as particularly interesting and valuable. We followed your advice and now cite the covariate-shift literature (Ross & Bagnell, 2010; Ross et al., 2011) together with conservative and trust-region policy updates (Kakade & Langford, 2002; Schulman et al., 2015) in Sec. 5.2 (p.8).

We also greatly appreciate your effort on explaining in detail where you see problems with our work, which, as we understood, are mainly about claimed scope and application domain. We'd like to respond to each point and are happy to explain how we have improved of them:

Critical Weaknesses:

  1. It is unclear why one might be able to easily construct a rule-based generator for tasks in general. For example, during a standard RLVR/RLHF pipeline, we fine-tune language models on tasks ranging from creative writing to question answering as a helpful assistant to mathematical reasoning, and for any of these tasks, constructing such rule-based generators may either be impossible or, at the very least, extremely expensive.

Thank you. We understand your point. Of course, our approach is hardly applicable to modern LLMs due to the fact that rule-based language processing works poorly. We are happy to improve our paper to be more clear about its scope. Thus, we revised sentences in the abstract, introduction, and conclusions, e.g. on p.2: "APA assumes an executable program whose outputs match the policy’s action dimensions. It therefore targets systems that start from a hand-written generator and are fine-tuned toward behavior its rules do not produce. It does not directly apply to token-level fine-tuning of language models, where neither such a program nor a factored action space exists."

We now define the scope as domains were rule-based-only approaches exist and are quite successful on its own, like music composition, but also control tasks (robotics, autonomous driving, smart buildings, etc.), PCG, game-playing agents, and similar. We would even say, that specific visual art generation and artful writing systems are worth a study. Overall, we see the music domain not as a limitation, but its dual tradition of rule-based vs. data-based approaches, both with many successful examples, as inspiration for the development of generic RL methodologies like APA.

  1. Continuing from point (1), on such more general tasks, it is unclear whether the effort spent on constructing such rule-based generators is better spent on directly addressing the reward specification problem by constructing more grounded reward functions (for e.g., refining rubrics heavily for language model judges or allowing adaptive rubrics to address the possibility that a model gets good along some dimensions while not improving/degrading along others).

We fully agree that constructing rule-based generators is expensive. We argue that research domains exist that already formulated rules and showed their success (again, not NLP, see previous comment). Enabling fine-tuning for such existing rule-based solutions is our main motivation. And yes, those rules may alternatively be instanciated as rewards, but we do argue in our paper why this way is more vulnerable to reward hacking as a reward patch can only fix the property it measures (see Fig.4). Further, rules in symbolic programs generate decisions directly, making them also more valuable in terms of interpretability.

  1. In addition, when such a rule-based symbolic program is available, I am not sure why naive DAgger could not work just as well as APA and a direct comparison against that would help the paper.

Thanks for this recommendation. As we understand it, pure DAgger uses no reward, so it would reproduce the program, with reward at the clone’s level (~0.40, the clone point in Fig. 2a). RL plus a DAgger-style imitation loss on the learner’s states is our c4, which APA beats by ~0.11 reward. We now explain this in Sec. 4, paragraph Conditions (pp.6--7): c4 combines RL with a DAgger-style imitation loss, and the clone mark in Fig.2a can be read as an approximation of pure DAgger. Related Work now makes the connection explicit as well (p.3). If you had a different setup in mind, please let us know.

AuthorsOpenReview ↗

Answer to Reviewer 15tt (part 2)

  1. It is unclear whether, for general tasks, one can factorize the action space into independent components the way the paper factorizes, say, the piano’s top note into pitch class, octave and volume. In particular, without this factorization, I am not sure that the anisotropic weights for KL penalties can even be instantiated. Insofar as the paper’s two main contributions are replacing the base model with a symbolic program generator and anisotropic anchoring, and since both of them seem impractical in general settings, I am not convinced of the practical usefulness of APA as an algorithm.

Thank you for this argument, although we think to not fully agree with it. Besides LLM fine-tuning, factored and multi-discrete action spaces are common and well established in RL. We provided examples for promising candidates for follow-up studies in other domains (control, PCG, etc.). PROPEL, CORE-RL, RUBICON and PPF showed that symbolic programs can successfully be used as priors in their domains. We do not quite understand why APA could be fully dismissed as practically unuseful. Does the scope correction help here?

  1. Furthermore, in the event that such factorization can be done, the existing approaches in the literature for better reward specification could potentially also work almost just as well.

To the best of our knowledge, preventing reward hacking in general is an open research problem. Do you have any better general reward specification method in mind that could potentially work here? Our general answer was Sec.5.4, which shows that adding another reward term repairs only the property "it measures" while the others keep degrading.

It is possible that the paper’s focus is on music generation tasks only and that can plausibly justify the assumption that accessing such a symbolic program is realistic. However, if that is the case, the paper should clarify so and not claim that it is addressing the general problem of reward misspecification in RL fine-tuning. The current framing causes the reader to expect that the algorithm is being proposed as an alternative in general RL frameworks.

Thanks again. As already said, we happily revised the scope accordingly. However, we confidently still think that APA is not restricted to music generation only, but, of course, testing APA's applicability to other domains remains an open research question that is high on our list for future work.

Questions For Authors:

Please see the section of critical weaknesses.

We hope the revised scope and the clarification on DAgger address your main concerns. Are there remaining points like imprecise sentences/presentations or missing analyses that would help your assessment? Thanks a lot, we really appriciate your efforts.

As the discussion period ends on 18 Nov, we wanted to ask whether our answers and the revised scope address your concerns. Thank you very much for your valuable feedback.

Reviewer_xiQXOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes APA, which anchors selected policy outputs to a symbolic program during reinforcement learning. In a jazz accompaniment task, it preserves harmonic competence while improving reward. Experiments compare anchoring schemes and show that querying the program helps when a frozen reference encounters chords absent from its training data.

Critical strengths
  • The comparisons across anchor shape, reference type, clone KL direction, initialization, and unanchored checkpoints provide useful evidence for the benefit of freeing the designated octave heads.
  • The paper reports where frozen-clone anchoring performs better, which helps establish when the executable reference is useful rather than presenting APA as uniformly superior.
Critical weaknesses
  • The main methodological contribution appears limited, as APA primarily extends the earlier scalar program anchoring with manually assigned per-head weights, while the theoretical analysis does not address the shared-network setting used in practice.
  • The evaluation uses a single music testbed with a reliable symbolic program and manually chosen target heads, leaving limited evidence for how well APA works with imperfect programs or less straightforward behavioral decompositions.
Questions for authors
  • How does APA compare with retaining the symbolic rules for protected outputs and training only the octave decisions, including the cost of running the program at inference?
  • Would the distribution-shift advantage remain with a transposition-augmented or root-equivariant clone?
4 comments
AuthorsOpenReview ↗

Answer to Critical Weaknesses

Thank you for your positive feedback. We were happy to read that you find our work useful and well scoped, that you learned something interesting, the text is clear and easy to follow, and that overall the paper appeared completely sound to you.

  • The main methodological contribution appears limited, as APA primarily extends the earlier scalar program anchoring with manually assigned per-head weights, while the theoretical analysis does not address the shared-network setting used in practice.

Fair point. We had thought that the manual assignment can be a feature when systems have an interpretable nature that can be exploited (like music systems and domains that are similar, like control tasks, PCG, etc.). Where it cannot, we'd argue that automatic tuning of the βk\beta_k is a straightforward approach. We will definitely integrate that idea to our follow-up studies. Thanks for poiting that out.

Overall, the contribution is more than the per-head weights, namely the separation of two choices the standard KL penalty fixes together. In distribution, the reference does not matter and the shape does (Tab.3); under shift, the shape and KL direction matters less and the reference does (Fig.3, the four clone conditions agree within 14%). This is now stated in the list of contributions (p.2).

About the theoretical analysis: We provided Proposition 1 as general motivation for the anisotropic anchor and acknowledged in Appendix A that the shared backbone will carry reward gradients into non-target heads. We provided Table 3 to measure that exact effect, and found that the anisotrophic anchors are holding even though parameters are shared (see c8 and c5). Appendix A now points to this measurement explicitly (p.15).

We also attempted a quick idea for a Proposition that includes the shared network, that could go like this:

Sketch: anisotropic anchoring with a shared backbone (first order)
Consider one state with input xx, a linear backbone h=Wxh = Wx, and softmax heads with logits zk=Ukhz_k = U_k h: a target head tt, and a held head jj that the reward does not read. Let gt=∇ztJg_t = \nabla_{z_t} J be the reward gradient on the target logits, and let the anchor on jj be the cross-entropy to the reference πjref\pi^{\mathrm{ref}}_j, whose gradient on zjz_j is β(πj−πjref)\beta(\pi_j - \pi^{\mathrm{ref}}_j). To first order in the step size η\eta, one gradient step on all parameters gives

Δzj=η[∥x∥2UjUt⊤gt−βKj(πj−πjref)],\Delta z_j = \eta\left[\lVert x\rVert^2 U_j U_t^\top g_t - \beta K_j (\pi_j - \pi^{\mathrm{ref}}_j)\right], Δzt=η[Ktgt−β∥x∥2UtUj⊤(πj−πjref)],\Delta z_t = \eta\left[K_t g_t - \beta \lVert x\rVert^2 U_t U_j^\top (\pi_j - \pi^{\mathrm{ref}}_j)\right],

with Kk=∥x∥2UkUk⊤+∥h∥2IK_k = \lVert x\rVert^2 U_k U_k^\top + \lVert h\rVert^2 I, which is positive definite.

  • Leak: without an anchor (β=0\beta = 0), the reward moves the held head through the first term of Δzj\Delta z_j unless the readouts UjU_j and UtU_t are orthogonal. This is why held heads need β>0\beta > 0.
  • Anisotropic anchor: treating gtg_t, UU and hh as locally constant, the held head settles where Δzj\Delta z_j vanishes (up to the softmax's shift invariance), i.e. at β(πj−πjref)=∥x∥2Kj−1UjUt⊤gt\beta(\pi_j - \pi^{\mathrm{ref}}_j) = \lVert x\rVert^2 K_j^{-1} U_j U_t^\top g_t, a drift of order 1/β1/\beta. Substituting this into Δzt\Delta z_t gives Δzt=ηStgt\Delta z_t = \eta S_t g_t with St=Kt−∥x∥4UtUj⊤Kj−1UjUt⊤S_t = K_t - \lVert x\rVert^4 U_t U_j^\top K_j^{-1} U_j U_t^\top, the Schur complement of the joint kernel of both heads. StS_t is positive definite, independent of β\beta, and St⪯KtS_t \preceq K_t: the target head is slowed by a fixed amount relative to no anchor, but never stopped. This is consistent with TopOct in Table 3 (drift 1.24 under APA vs. 1.53 unanchored).
  • Scalar anchor: additionally adds −βKt(πt−πtref)-\beta K_t(\pi_t - \pi^{\mathrm{ref}}_t) to Δzt\Delta z_t, which grows with β\beta and pulls the target head back to the reference, so the trade-off of Proposition 1 reappears.

We have not fully thought this through (therefore we call it a sketch). However, would you find it highly valuable if we worked this out formally for the revision (maybe including: expectation over states, nonlinear backbones in the linearized regime, and held heads that the reward reads indirectly)? If yes, we could attempt to work something like this out for the Appendix A until the rebuttal deadline. Thank you for your opinion and suggestion.

  • The evaluation uses a single music testbed with a reliable symbolic program and manually chosen target heads, leaving limited evidence for how well APA works with imperfect programs or less straightforward behavioral decompositions.

As Reviewer PQyJ also asked for experiments with "imperfect programs", we used the recent days to implement some. They also touch the problem of less seperable behavior. We will discuss the experiment in a separate comment as soon as results arrive.

AuthorsOpenReview ↗

Answers to Questions For Authors

  • How does APA compare with retaining the symbolic rules for protected outputs and training only the octave decisions, including the cost of running the program at inference?

This is a particularly interesting question. We have implemented experiments on that, which are now running. We will report their results as soon as they arrive. They will supplement Fig.2(a). Design and results will again be posted in a separate comment, as Reviewer PQyJ also asked for that.

  • Would the distribution-shift advantage remain with a transposition-augmented or root-equivariant clone?

We are confident to assume that the effect remains. Surely, the root part of the shift would vanish. However, the advantage on unseen chord qualities would remain, showing a smaller but consistent effect. Is that explanation sufficient for you? If not, we are happy to run those experiments after the hybrid and imperfect-program experiments and report it if it finishes in time.

As promised, we ran the experiments, see the comment below titled "Transposition-augmented clone (Q2)"

Results for your Q1 (hybrid) and W2 (imperfect programs) are now posted in our comment “Results of both additional experiments have arrived!” and in the revised PDF (Sec. 5.1 and Fig. 2a; Sec. 5.3; App. C.4; App. D.7). We would be glad to hear whether they address your questions, and whether you would like us to work out the shared-backbone sketch.

AuthorsOpenReview ↗

Transposition-augmented clone (Q2).

Transposition-augmented clone (Q2). As announced, we ran this experiment. It is now in the revised PDF (Appendix D.2, Figure 8, referenced at the end of Section 5.2). We refitted the shift clone on its training tunes that we transposed into all 12 keys, so that it sees every root and only the chord qualities remain unseen, and reran c5 and c8 (rev.) under shift over the full β\beta sweep with five seeds.

The result matches Fig.2b as expected. Augmentation raises the clone's competence from 0.603 to 0.770, as the root part of the shift vanishes, but the clone-anchored condition again levels off at its reference (0.77 for every β≥0.1\beta \geq 0.1), whereas the program-anchored condition reaches 0.942 at β=0.3\beta=0.3 and 0.969 at β=1\beta=1, at the same reward (0.736 vs. 0.739 at β=0.3\beta=0.3). The competence gap halves (0.17 instead of 0.33). What remains is consistent with the chord qualities the clone never saw, which root augmentation alone cannot supply.

Thank you again for raising this question. We think the result, with the new Figure 8, really makes a nice addition to our study.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To all reviewers: Additional experiments

The following two additional experiments are currently running. We will report results as soon as they arrive (we expect 15 or 16 Nov). are finished, see comment below this comment.

Hybrid policy -> training only the octave decisions
Plan: The program plays the held heads, RL trains the octave heads, 5 seeds.
This setting completes the sweep over β\beta. Conceptually, it sets βk=∞\beta_k=\infty for the held heads. Practically, we erase the PPO loss term from Eq.(2) and fully release the shared network dependence from the targeted heads. The result will be used to complete Fig.2(a) and report asap here. Running the program costs one rule evaluation per step, which is negligible here; the hybrid needs the program at inference, whereas APA yields a standalone policy.

Imperfect programs
Plan: corrupt one held head vs. corrupt one target head.
Held head: we corrupt some of the chord qualities by deterministically constructing wrong voicings, yielding to errors with constant error rates. Target head: with a constant error rate, the octave is corrupted by uniformly sampling a different one. For both settings, we measure how the error rates are affected by APA fine-tuning.

AuthorsOpenReview ↗

Results of both additional experiments have arrived!

Hybrid policy (xiQX Q1, PQyJ Q1)

We are happy to report, that we were able to add the requested hybrid as condition c9. The program executes all held heads, and RL trains only the two octave heads, with 5 seeds and the same protocol as the other conditions. Paired by seed, it has +0.016+0.016 competence and −0.008-0.008 reward relative to APA at β=0.3\beta = 0.3, and lies above APA at β=1\beta = 1 in both coordinates. It is therefore the end point of APA's frontier: holding the non-target dimensions outright performs like APA's strongest anchors, but offers only this one point. APA reaches the same operating point with a single standalone policy, and its coefficients βk\beta_k can also trade competence for reward along the frontier, which the hybrid cannot.

  • Results: see Section 5.1 and Fig. 2a (star).
  • Shared-network effect (xiQX): see Section 5.3 and Table 3.
  • Implementation and inference cost (xiQX): see Appendix C.4. The hybrid is faster at inference here (1.71.7 vs. 2.72.7 ms per beat).

Imperfect programs (PQyJ Q2, xiQX W2).

We deliberately corrupted the piano program in two ways, refitted the clones, and reran c1, c4 and c5 over the full β\beta sweep with five seeds (see new Appendix D.7, Figure 11, Tables 10&11; summary at the end of Section 5.3):

  1. Corrupt held dimensions (m7 chords read as dominant sevenths, one wrong note on 23% of beats): APA inherits the error at the program's own rate (99.2% vs. 98.9%) without amplifying it (no additional wrong notes), with unchanged reward (0.736 vs. 0.739 at β=0.3\beta=0.3). The policy's errors are therefore predictable: they are exactly the program's, which can be read off its rules, unlike those of a learned reference, which only testing reveals.
  2. Corrupt target dimension (top-voice octave randomized on 20% of beats): the error does not reach APA (reward 0.735 vs. 0.739; large register jumps 0.004, as without the error), whereas the scalar anchor copies it and loses more reward the stronger it anchors, up to 0.063 at β=1\beta=1. The designer therefore does not need to get the program right on the dimensions the reward is meant to change, which a scalar anchor would require.

We are very happy with this result and thank the reviewers for pointing this out to us.


An additional insight on Inseparable behaviors (xiQX W2):

The decomposition in our testbed is already not clean: the reward scores absolute pitch, which spans the held pitch-class heads and the free octave heads, and it reads the piano's bottom note, which the held inner voices set (Table 4). In addition, all heads of a module share a network, so βk=0\beta_k=0 does not free a head entirely (Section 5.3). However, APA works despite this overlap, because the coefficients separate what the reward should change, not what it reads. Where a behavior cannot be split across heads at all, its head carries a single coefficient, and APA reduces to a scalar anchor there, but is not falling below it.


All changes are marked in blue in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.