acceptodds
Under review as a conference paper at ICLR 2027

Learning Reusable Spaces of Feedback Policies

Abstract

Learning policies that combine diverse skills remains a challenge for robotics and embodied systems, particularly when a controller must reuse and adapt learned behaviors across new tasks. Existing approaches learn latent or task-conditioned policies and behavioral priors for diverse skills, but they do not explicitly learn a shared functional representation of the underlying feedback policies. We present a method for learning a low-dimensional space of feedback policies, where each policy is represented by a small coefficient vector, enabling identification, synthesis, interpolation, and analysis. Specifically, we learn neural basis functions from expert trajectories to span a low-dimensional subspace of feedback policies. The linear coefficient structure reduces policy identification to coefficient estimation, policy synthesis to low-dimensional optimization, and interpolation to direct operations in coefficient space. Across humanoid, quadruped, and planar-quadrotor control, the learned space identifies held-out policies from sparse demonstrations, supports synthesis and interpolation, and enables faster, more accurate adaptation than prior methods while maintaining competitive closed-loop performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
robotics, multi-task control, policy representations, learning-based control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oZEDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies function encoders as a representation of feedback-policy families. It learns shared, vector-valued neural basis functions from expert state-action datasets and represents each policy by a small coefficient vector, typically with eight entries. Once the basis is frozen, the coefficients can be estimated from a demonstration by ridge regression, optimized against a new control objective, or interpolated between policies. The appendix derives coefficient sensitivities and maps coefficient covariance into action and approximate closed-loop trajectory covariance. Experiments cover command-conditioned and motion-conditioned G1 humanoid policies, Go1 locomotion, a planar quadrotor, and LQR, with additional pendulum and HalfCheetah comparisons. The strongest results concern sparse identification within well-covered behavior families, including improvements in Go1 tracking and survival. The paper also reports objective-only synthesis and local interpolation, while documenting failures when behavior coverage is insufficient or interpolated behaviors are distant.

Critical strengths

1 Useful sparse identification with meaningful closed-loop gains

The shared vector-valued basis is a sensible way to couple action dimensions while keeping online adaptation small. It can also represent independently trained experts without aligning their network parameters. G1 coefficient identification takes 0.675 ms on CPU once the 128 context pairs are available, supporting inexpensive inference after observation.

2 Broad evaluation with explicit reporting of limitations

The paper evaluates held-out trajectory prediction and fresh-reset execution, includes disturbed resets, and distinguishes methods with access to sampled demonstrations from methods with access to complete policies. It examines capacity, context size, offline coverage, interpolation, synthesis, and different teacher constructions. The authors also report unfavorable results, including omitted-family failures and unstable interpolation, instead of presenting the linear span as a universally reliable composition mechanism. This makes the study informative and gives useful context for its favorable identification results.

Critical weaknesses

1: Methodological Novelty Is Insufficiently Differentiated From Prior Work

MoS-VLA already represents robot policies as linear combinations of learned vector-valued basis functions with coefficients inferred from demonstrations — the core construction of Eq. 2–4. The distinction that this paper uses coefficients for control behaviors after removing command inputs is relevant but narrow. Directly representing a policy in a learned function space is established prior art, and the paper does not sufficiently isolate what is genuinely novel beyond this design choice.

2: The Key Baseline Is Structurally Incapable of Using Demonstrated Behavior, Making the Comparison Uninformative

The static PULSE-style comparison model largely ignores the variable intended to distinguish different behaviors, meaning different demonstrations do not lead to meaningfully different recovered behaviors. Outperforming this baseline on behavioral accuracy therefore does not establish an advantage over a properly functioning nonlinear alternative. A valid comparison requires first verifying that the baseline actually recovers distinct behaviors from distinct demonstrations, then comparing tracking, task completion, and smoothness under identical conditions — none of which is provided. The paper lacks real-world evaluation.

Questions for authors

Major questions:

  1. The zombie-walk-to-jog interpolation succeeds in only 22 of 56 trials (39.3%), yet no analysis is provided on when mixing is expected to work or fail. Can the authors provide more behavior pairs and clearer criteria for predicting interpolation reliability?

  2. The participation rank of 5.58 is derived from the eigenvalue spectrum of an occupancy-weighted metric, but whitening the basis makes any positive-definite metric the identity, yielding rank eight without changing any represented policy. How do the authors establish that this metric measures an intrinsic effective dimension rather than an artifact of basis choice?

  3. The main Go1 split reuses the same 28 policy functions across offline learning and testing, and the headline G1 unseen-command test covers only two commands and 12 rollouts within the coordinatewise training range. Does the evaluation establish genuine generalization beyond the offline policy support, and can broader out-of-distribution conditions be tested?

  4. All evaluations are conducted in simulation with no physical robot deployment reported. Can the authors provide real-world results, or at least discuss what sim-to-real gap is expected given the method’s reliance on learned basis functions?

Minor questions: 1 How well does the method work on behaviors absent from training?

Please separate tests of new trajectories from tests of new behaviors. Beyond the existing held-out command and transfer tests, can you evaluate more omitted behavior families on the legged robots and report both tracking accuracy and episode completion across independent training runs?

2 When can two behaviors be mixed successfully?

Can you evaluate more motion pairs, explain why some mixtures fail, and give a practical indication of which pairs are likely to work? The difference between 96.4% and 39.3% success makes the limits of this operation especially relevant.

3 When is searching better than selecting an existing policy?

Please compare search with selecting the best stored coefficient on several new targets. Can you report the total task score, every target component including Go1 height, the number of rollout evaluations, and elapsed search time? This would clarify both the benefit and its cost.

4 Can the nonlinear comparison model use the demonstration effectively?

Can you adjust the static PULSE-style model or add another nonlinear comparison that demonstrably uses the supplied examples to distinguish behaviors? Please evaluate that model with the same data and report task completion, tracking accuracy, and smoothness.

5 How much of a demonstration must be observed before execution?

What happens if the model receives only the first one or two seconds, rather than samples spread across the full trajectory? Please report the observation duration and computation time alongside the resulting execution quality, so the total adaptation time is clear.

6 Which model configuration produced the main comparison results?

Please reconcile the architecture, parameter count, and training-update count in Appendix B.2 and Table 19. Which exact configurations and how many independent training runs were used for Tables 1 and 20?

4 comments
  • (W1) Please see our response to Reviewer XFdj, W1. Removing the command input makes the coefficient the task, and the state-feedback form lets one set of coordinates support identification, objective-driven synthesis (Eq. 9), interpolation, and closed-loop uncertainty (Eqs. 12 to 15). MoS-VLA's image-to-action basis has no state dynamics, so it supports neither synthesis nor uncertainty, and no prior function encoder work demonstrates these.
  • (W2, Minor 4) The static PULSE-style model is our own construction, built for an information-matched comparison (same 64 state–action pairs, one fixed latent per rollout). Nothing in its structure forces the latent to carry behavior information, so the decoder can ignore it and fit one pooled controller. This collapse is expected, and it is a result, not a miscalibration (App. B.9). The FE avoids it because the policy is linear in the coefficients. The real baseline is the dynamic PULSE model, which follows the original paper. Its latent does carry behavior information, yet it reaches 87.7% success versus 97.7% for the FE, with nearly twice the jerk, despite seeing the reference at every step. We also compare against other baselines including FOMAML, a nonlinear model that uses the demonstration and that the FE outperforms in action error and tracking (Table 1), plus a ridge head, pooled BC, a random subspace, functional PCA, nearest-expert retrieval, a stored-policy simplex, and a parameter subspace (Gaya et al., 2022).
  • (Q1, Minor 2) Mixtures carry no intrinsic stability guarantee (as we say in Sec. 4.3). The span is a set of System 0 behaviors, and a higher-level layer would schedule coefficients that keep rollouts stable. Across all 21 pairs of the 7 ProtoMotions behaviors, interior success averages 72.7% (range 21.4% to 100%), and neither endpoint distance nor an endpoint-residual proxy predicts which pairs mix well (Spearman −0.24 and −0.22). We will add these results.
  • (Q2) Thank you for identifying this error. The participation rank of MM depends on the choice of basis. The invariant quantity is the spectrum of MΣcM\Sigma c, the metric-whitened PCA we already report (4 directions, 95.25%), and we will use it instead.
  • (Q3) Our claims concern policies within the learned span. Within that scope, we see generalization to held-out behaviors close to the span. On 32 held-out quadrotor tasks, identified policies reach 65.1% success against 64.4% for the analytical teacher (Table 10). On held-out LQR objectives, cost is within 0.002% of the Riccati optimum. On G1, all 56 rollouts on 14 held-out commands survive (Table 5), and held-out action RMSE matches training (0.0438 vs. 0.0427, Table 1).
  • (Q4) Real-world experiments are outside the scope of this paper, and we will discuss this in our limitations. Our approach does not address the sim-to-real gap or confer robustness beyond standard simulation-trained policies. The basis can be learned from real trajectories, but deployment would require stronger domain randomization, more data, DAgger, and System 1 behavior synthesis.
  • (Minor 1) Our evaluations separate new trajectories from new behaviors (Tables 1, 5, and 10; App. B.5). Table 13 removes each of the four quadrotor families in turn, and on Go1, removing curved walking from training drops its survival from 0.974 to 0.320 (Table 9), as expected. We will add omitted legged families across independent training runs.
  • (Minor 3) Retrieval suffices for seen behaviors; search matters for unseen ones. On five held-out Go1 targets, search takes as little as 3.1 s (432 rollouts) with a small loss in performance (mean CMA-ES score −0.349 vs. −0.330 at the full budget), and about 80 s (14,112 rollouts) at the paper's budget. We will report the full ablation, with all error components, rollout counts, and timing.
  • (Minor 5) Using only the first 1 or 2 s of a demonstration gives identification accuracy, survival, and tracking comparable to samples spread across the trajectory (see the appendix sample-size sweeps). The linear solve takes 33 to 38 μ\mus, and full identification 0.63 to 0.96 ms. We will add these results.
  • (Minor 6) We will fix the mismatch between App. B.2 and Table 19. In Table 1, the G1 joystick and quadrotor rows use a 2$\times$256 SiLU basis over 3 seeds, and Go1 over 4. The ProtoMotions rows (Tables 1 and 20) use a 3$\times$512 SiLU basis (0.678M parameters) trained for 250,000 updates. We will state configurations and seeds in each caption.

Based on the reviewer’s comment, we retrained the static PULSE baseline with β=0\beta=0 to test whether KL regularization contributed to latent collapse. Removing the KL term restored demonstration dependence: swapping demonstration latents increased held-out action RMSE from 0.01582 to 0.25365. In a matched retraining with the original β\beta schedule, the error remained approximately 0.01717 with either the correct or swapped latent, indicating that the policy largely ignored the demonstration.

Removing the KL term eliminates the training penalty that pushes the inferred latent distribution toward a standard normal prior. This allows the model to encode behavior differences without paying that regularization cost. The resulting model retains the same static encoder–decoder architecture and provides a stronger nonlinear comparison. From common physical resets, its demonstration latents recovered the intended behavior in 30/35 trials, compared with 5/35 for the matched KL control and 32/35 for FE, according to our diagnostic. Across 700 paired closed-loop trials, FE achieved higher completion (98.0% versus 95.3%) and slightly lower behavior-descriptor error (0.208 versus 0.214). However, the corrected PULSE baseline achieved slightly lower teacher-action RMSE on visited states (0.168 versus 0.169) and a lower action-change rate (5.34 versus 5.55). The corrected comparison therefore shows similar action accuracy and a completion advantage for FE, alongside a modest smoothness advantage for static PULSE. FE’s closed-form coefficient identification remains a separate methodological benefit.

We will revise the PULSE results to distinguish the original KL-regularized model from the β=0\beta=0 model, include the demonstration-swap diagnostic, and report completion, tracking, and smoothness for both. We will also revise the discussion to acknowledge that latent collapse weakened the original baseline and qualify the remaining performance differences.

Thanks to the authors for the corrected PULSE baseline (β=0) and for acknowledging the participation-rank error. Three concerns remain unresolved. 1) The manuscript's novelty over prior work (manuscript Linnes 101–106, page 2; Eq. 2–4, page 3): the core construction \textbf{neural basis functions with least-squares coefficient estimation} is already established in function encoders (Ingebrand et al., ICML 2025) and applied to robot policies in MoS-VLA; synthesis, interpolation, and uncertainty follow naturally from that same linear structure, and the response does not isolate a genuinely new algorithmic contribution beyond this. 2) is corrected PULSE comparison (authors comment on 16 Nov): the β=0 baseline is now a properly functioning nonlinear model, yet the FE’s advantage is only a 2.7% completion gap (98.0% vs 95.3%), matched action accuracy (0.168 vs 0.169 RMSE), and a smoothness disadvantage (action-change rate 5.55 vs 5.34), confirming the original comparison was uninformative and that the advantage over a working nonlinear baseline is modest. 3) Interpolation reliability (manuscript Page 8, Sections 4.3; authors’ Q1 response): the 39.3% zombie-walk-to-jog success remains unexplained; however, the authors confirm neither proxy predicts which pairs mix well (Spearman −0.24 and −0.22), and real-world evaluation (I cannot see any value for real-world validation) was explicitly declined. I still maintain my rating.

We thank the reviewer for their feedback. We maintain that the contribution does not follow automatically from the shared linear formulation. A natural operation does not automatically establish a capability, and none of the demonstrated capabilities in our work are investigated in prior art. Our central contribution is demonstrating that one learned feedback-policy basis supports sparse identification from expert demonstrations, runtime synthesis of behaviors from task objectives, and interpolation, together with our analysis of policy sensitivity and identification uncertainty. These operations reuse the same representation without retraining the basis. The prior work establishes the underlying construction, but does not investigate this.

The corrected PULSE results show that a nonlinear latent model can achieve comparable performance on this task. This narrows the performance claim for that comparison, but our approach remains the only one that supports the combined capabilities we investigate.

Reviewer_XFdjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

the paper proposes to represent a collection of expert policies using a small set of learned basis functions. they train the basis from expert demonstrations so that each policy can be represented by just a few coefficients. for new tasks, they either estimate these coefficients from demonstrations or optimize them directly using the task reward without updating the basis. they test this on humanoid and quadruped locomotion, planar quadrotor, and lqr, and also study policy interpolation and the structure of the learned policy space.

Critical strengths

the paper has a fairly thorough set of experiments across different control problems. i also appreciate the effort to study the learned policy space itself, including its effective dimensionality, where interpolation works or fails, and the uncertainty analysis

Critical weaknesses
  1. limited methodological contribution: the core method seems to largely follow the existing Function Encoder framework and it already learns neural basis functions and estimates coefficients via least squares. MoS-VLA has also applied this idea to robot policies. this paper extends the approach to a se t of different control problems, and further explores coefficient-space policy search.however, i didn't find a substantial new contribution from reading the intro or related work sec
  2. limited scientific insight beyond demonstrating the capabilities of the representation. the overarching claim seems to be that a learned low-dimensional policy space can support identification and synthesis of new behaviors. while the experiments show these capabilities, they are largely natural applications of the chosen representation. i find it difficult to identify a more general scientific finding that comes out of these experiments. for instance a more interesting question would be whether useful feedback policies actually lie in an approximately low-dimensional linear space, and under what conditions
  3. limited characterization of when the learned policy space works. the paper shows that the learned basis can recover expert policies and synthesize some new combinations of behaviors. the authors also examine basis dimensionality.but it's still unclear what determines whether a family of policies can be represented well by a small linear space. the LQR results are also not particularly convincing in this regime, since the optimal controllers already belong to a low-dimensional linear function space. overall, i don't think the experiments provide much understanding of the conditions under which this representation works well or starts to fail.
  4. limited evidence beyond simulation. all experiments are conducted in simulation with the policy basis learned from demonstrations generated by existing expert controllers. it remains unclear how well the learned policy spaces would work on physical robots
Questions for authors
  1. could the authors clarify the main methodological contribution relative to function encoders and MoS-VLA? both already learn neural basis functions and estimate coefficients from demonstrations. what specifically goes beyond these prior methods? i feel this is still rather unclear after reading the intro and related work.
  2. can the authors more systematically investigate the dimensionality of the underlying expert policy families? for instance, how does the required number of basis functions scale with the number and diversity of expert behaviors? can the authors also better characterize which new tasks can be represented by the learned space, and when optimizing within that space is better than adapting the full policy network under comparable compute budgets?
1 comment
  • (W1, Q1) We build on the structure of function encoders (Sec. 3), which have been used to represent spaces of dynamics and of VLA policies (MoS-VLA). Our contribution is a learned space of feedback policies that supports identification, synthesis, interpolation, and closed-loop uncertainty. Synthesis, interpolation, and uncertainty are new, and we also characterize the learned space of behaviors itself. See our response to Reviewer oZED, W1 for a fuller comparison with MoS-VLA.
  • (W2) We think the experiments do answer the question you raise. The G1 and Go1 locomotion policies sit close to an 8-dimensional learned span, and 4 directions carry 95% of the G1 policy variation. The space is locally convex in closed-loop behavior, but not globally. And what matters is coverage of the demonstrated states, not the dimension: a random 8-D subspace gets 32.6% success versus 65.1% for ours. We will make these points explicit in the paper.
  • (W3) There are two parts here: when a set of behaviors is well represented by a span, and whether we can tell for a given set of behaviors that such a span exists. For the first, our experiments point to coverage. The span represents a behavior well when its state–action pairs lie near the span on the states seen in the demonstrations, and it fails when they do not, as with omitted families (Go1 curved-walk survival drops from 0.974 to 0.320) and distant interpolations (39.3% success). For the second, the projection residual and the capacity sweeps (Tables 5 and 9) measure this after training, but we do not have a test that predicts it beforehand, and our experiments do not answer that in general. For linear systems, our interpretation aligns with theory: LQR gains form a linear family, and the learned span recovers them to within 0.002% of the Riccati cost.
  • (W4) Hardware experiments are outside the scope of this paper. Identification only needs state–action pairs at test time, so nothing in the method is tied to simulation.
  • (Q2) There is no general rule that n behaviors need k basis functions, since that depends on how the behaviors relate to each other, not just how many there are. What we provide is a way to find k for a given set of behaviors or tasks after training. The capacity sweeps (Tables 5 and 9), the metric-whitened PCA of the learned coefficients (4 directions carry 95% of the G1 policy variation), and the spectrum of the policy kernel (Sec. 4.3) show how many directions the data actually use. On searching the space versus adapting the full network under a matched compute budget, the answer depends on the target and the budget, so a single tradeoff curve would not be very informative. What our experiments do show is that with a demonstration, fitting the coefficients beats full-network FOMAML adaptation on Go1 (action error 0.052 vs 0.062, survival 0.978 vs 0.878, 4.4$\times$ faster), and that without one, full-network adaptation only has an advantage when the target is outside the span (Table 13).
Reviewer_nA79OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors present the work as a method for learning a low-dimensional space of reusable feedback policies from expert demos. They use neural basis functions to represent different policies through a small set of coefficients, in this way, it allows new policies to be identified from limited state-action observations or synthesized by optimizing these coefficients. The learned representation also supports policy interpolation and uncertainty analysis without needing to retrain the basis functions. The method is evaluated on envs such as humanoid, quadruped, planar quadrotor, etc control tasks, and results show good results in policy identification, adaptation efficiency, and closed-loop control in comparision to several baselines.

Critical strengths
  1. I think the idea of learning a reusable feedback policy space is interesting. Instead of fine-tuning a large policy for each new task, the authors reduce adaptation to estimating a few coefficients. This is especially valuable for real-time control, where adaptation efficiency is important.

  2. The learned space is not only used for policy identification, but also supports policy synthesis, interpolation, and uncertainty analysis. This makes the framework more meaningful than simply learning a low-dimensional latent representation.

  3. The authors evaluate the method across humanoid, quadruped, and quadrotor control, with comparisons against several relevant baselines. The closed-loop evaluations is useful, since low action prediction error alone does not necessarily guarantee good control performance. Such experiments provide evidence that the learned policy space can capture reusable control behaviors.

Critical weaknesses
  1. My main concern is the assumption that diverse feedback policies can be represented by a small number of shared basis functions. This might work for closely related locomotion behaviors, but I am not fully convinced it can generalize to more complex tasks with fundamentally different control requirements. As the diversity of behaviors increases, the underlying policy space may become highly nonlinear or even disconnected, where I feel the single linear subspace is no longer sufficient.

  2. The method heavily relies on expert trajectories to construct the policy space, and the learned basis is frozen during adaptation. This means that new policies are fundamentally restricted to the behaviors that can be expressed by the existing basis. I think this can be a limitation, especially when new tasks require control capabilities not covered by the original demos. The unsuccessful interpolation cases and degradation when certain behaviors are excluded from training already suggest this issue. It is unclear whether the method can truly learn new behaviors or we should say it recombines existing ones.

  3. Another is that the learned basis functions are tied to a specific state-action space, making the reusable policy representation difficult to extend to systems with different observation or control structures. I think a linear combination of individually effective feedback policies does not necessarily produce another stable policy. Even small errors in the learned representation may accumulate through closed-loop interactions and lead to unstable behaviors. This raises a fundamental question about whether the proposed coefficient space can reliably support policy synthesis and adaptation beyond the relatively constrained settings considered in this paper? I would like to see author's opinions on that.

Questions for authors
  1. Could the authors discuss how the required number of basis functions changes as task diversity increases? In particular, is there any way to measure whether a new task falls outside the learned policy space, and determine when additional basis functions are needed?

  2. The paper shows some performance degradation when certain behaviors are excluded from training. I wonder whether the authors can explain the boundary between synthesizing new behaviors and recombining existing ones? E.g., can the method handle a new control requirement that is not represented in the expert demos, or would it necessarily require updating the learned basis?

  3. Since the method constructs policies through linear combinations of neural basis functions, what properties (if any), ensure that the synthesized policies remain stable? Can the authors provide more analysis of when coefficient interpolation leads to unstable behavior, particularly for more distant policies?

  4. I think this work is closely related to some existing research on reusable policy representations and efficient adaptation. For example, [1] explores latent-space adaptation with frozen foundation policies, [2] investigates low-dimensional policy subspaces for online adaptation, and [3] introduces function encoders for representing and transferring functions in Hilbert spaces. While these methods have different formulations, they share similar motivations. Could the authors clarify the main methodological differences, particularly what the proposed function-space representation enables beyond previous latent-space and parameter-space adaptation methods?

[1] Da et al. Latent Adaptation of Foundation Policies for Sim-to-Real Transfer. ICLR 2026.

[2] Gaya et al. Learning a Subspace of Policies for Online Adaptation in Reinforcement Learning. ICLR 2022.

[3] Ingebrand et al. Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces. ICML 2025.

1 comment
  • (W1) Generalizing to more complex tasks needs more than a single feedback policy. Our approach sits at the System 0 level, meaning each coefficient vector is one low-level feedback behavior, like walking, turning, or crouching. More complex tasks like pick-and-place or getting up after a fall mean chaining simple behaviors, which a higher-level layer can do by scheduling coefficients or optimizing them online. This is exactly what our representation enables, and it becomes efficient because of the coefficient structure. Since the space is linear, switching or blending behaviors is just a change to a k-dimensional vector, and identifying a new behavior is a single least-squares solve (0.675 ms on G1), not a network update. The same coordinate structure supports synthesis by optimizing those k numbers directly (Sec. 4.2), so a higher-level layer searches over a lower-dimensional coefficient vector instead of optimizing network weights.
  • (W2) Yes, the learned policies are limited to the span of the behaviors seen during training. However, we show that interpolation and synthesis within the span can produce behaviors that are not in the training data, like G1 commands that mix forward, lateral, and turning motion, or the Go1 crouched turn. Nevertheless, there is no guarantee these are perfectly stable, as the zombie-walk-to-jog interpolation shows. At the System 0 level this is expected: a higher-level layer would schedule coefficients that keep rollouts stable (see Q3).
  • (W3) The basis is tied to a particular state–action definition, the same as any policy. However, these are closed-loop feedback policies, not task-conditioned ones, and closed-loop stability does not need a perfect fit: the identified G1 policies finish 97.6% of full rollouts even with nonzero action error. To your question about reliably extending to meaningful complex behaviors, our own separate testing shows that scheduling coefficients enables complex stand-up behaviors that are not seen during training when the basis is trained on a larger subset of the BONES-seed catalog and evaluated on get-up tasks from random restarts and falls. In this paper, we are mainly concerned with showing the capabilities, characteristics, and qualities of the learned policy space on a controlled set of system 0 behaviors.
  • (Q1) We usually only know how many basis functions we need once we have the data, and more task diversity generally means more basis functions. For our locomotion tasks, 8 is enough for both G1 and Go1 (Tables 5 and 9). Yes, there is a way to measure whether a task is well represented by the policy space. Because the space is linear and we use least-squares, the projection residual on the context data acts as a metric that flags tasks that fall outside the span and could indicate when training additional basis functions would be needed.
  • (Q2) Novel behaviors outside the span are not generally supported. We would not expect the robot to jump after training only on walking gaits. Our experiments do show that it can synthesize mixtures and combinations of the gaits it has seen.
  • (Q3) There is no explicit stability guarantee. A mix of stabilizing controllers is not stabilizing in general, even for linear systems. In practice it mostly comes down to how well the policies are distilled: nearby interpolations succeed (95/96 rollouts on G1), and failures show up on paths through regions the data covers poorly (39.3% for zombie walk to jog). In general, we do not expect every point in the span to be stable. The span is the set of available System 0 behaviors, and in practice a higher-level layer would choose among them by scheduling coefficients that keep rollouts stable according to the task metric.
  • (Q4) We cite [2] and [3] and compare against [2] directly (Sec. 4.1, App. B.8). [3] is the function encoder algorithm we build on (Sec. 3). [1] also fits a latent with least squares, but its policy is nonlinear in that latent, so it needs an adapter network and does not give closed-form identification, synthesis, or uncertainty. It also targets sim-to-real dynamics shift, not identifying or combining behaviors. We will add it to the related work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.