- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The main aim of this work is to formalise and propose a new benchmark based on resource allocation in dynamic environments, where there are multiple types of constraints and deadlines. One of the contributions is to present this problem in an environment where tasks and constraints may change over the course of the problem, turning it into a continual learning problem.
Critical strengths
From what I understand of the work, its main potential contribution lies in the formalisation and creation of a new benchmark. I am not familiar with prior work in this specific area, but I imagine this benchmark could fill a gap in existing benchmarks. Creating a comprehensive benchmark is no easy task and usually requires ongoing feedback in the community.
Critical weaknesses
The biggest weakness lies in the writing, which makes it difficult to understand the depth and details of the work. There is no clear motivation, no identification of the gap this work seeks to fill (Why is this benchmark important?), and no clear description of the problem or the benchmark presented. Perhaps the work contains a very good idea and has valuable contributions to offer the community, but the current version is unable to convey it properly, and that contribution may be lost.
Questions for authors
- What gaps does this work aim to fill?
- Could you provide a simple and specific explanation of the problem you are seeking to solve? Let’s leave the mathematical aspects aside for the moment.
- Concerning the Continual Learning aspect, what sort of distribution shift do you encounter in this problem? What sort of forgetting might the model experience? What sort of knowledge transfer would be beneficial?
- For the two families of benchmarks presented: what is the relationship between them, and what motivations and gaps are they seeking to address?
4 comments
Q3. Concerning the Continual Learning aspect, what sort of distribution shift do you encounter in this problem? What sort of forgetting might the model experience? What sort of knowledge transfer would be beneficial?
In the deployment setting described in Q1–Q2, continual task adaptation and external resource scheduling create two interacting demands on the same model.
Distribution shift. The problem faced by the deployed service changes over time as operating conditions or the mix of requests change. Consequently, recent observations may follow a different task distribution from earlier ones, and the model needs to adapt using incoming feedback. The cluster's allocation can change independently because of other applications. A capacity change affects the learner's available representation. The setting combines a changing task distribution with an externally changing deployment constraint.
In the linear family, members of a group share input directions but use different coefficients to combine them into a target. For the same input, members A and B therefore generally require different outputs and have different joint distributions of inputs and targets. Switching groups also changes the shared input subspace and the active members, requiring adaptation to another set of linear prediction rules.
In the nonlinear family, members of a group share nonlinear features and combine them using different coefficients. Changing members changes the nonlinear function currently being predicted. Changing groups also changes the input projections and nonlinear responses that generate those features, so the change involves both the shared features and their combinations.
In both families, each member's target rule remains fixed, and the input sampling distribution stays the same across task changes. Members are sampled with fixed probabilities within a group. Thus, members have different conditional task distributions, while the overall mixture remains stable during one group visit. Group changes alter the active member set and its mixture over time, and earlier groups subsequently return. Resource changes are generated independently and can occur within a group visit or coincide with a group change.
Forgetting. Learning from recent data can interfere with knowledge useful under earlier conditions. Shrinking to satisfy a resource request can additionally remove parameters carrying useful knowledge. The controller needs to consider how continual learning and capacity changes jointly affect the model's existing abilities.
Specifically, we ask whether error on a previously learned task rises after subsequent learning or shrinking, although its target rule and input distribution remain unchanged. The default learner trains candidates using a bounded replay buffer. As new samples replace earlier-group samples, subsequent training places fewer constraints on earlier tasks; once a candidate is deployed, this can appear as worse performance on those tasks.
In the linear family, each member has its own output coefficients, but predictions also depend on a low-rank basis shared by all members. Updating this basis while learning new members or groups can change an earlier member's predictions even if its coefficients are retained. When the directions needed by earlier and current groups exceed the available basis capacity, adapting to the current group makes retaining earlier directions harder. Shrinking can directly remove some of those directions.
In the nonlinear family, members have different task embeddings but share the hidden layers and output layer. Learning the current group updates these shared weights and can change the nonlinear responses used by earlier tasks. Adjusting existing hidden units for a new group can impair earlier predictions; shrinking additionally removes hidden units and connections and can cause further loss of learned features.
When an earlier group returns, prediction error and recovery under the same task distribution allow knowledge retention to be assessed. The setting therefore includes forgetting caused by continued updates at a fixed model size, together with additional knowledge loss caused by externally required shrinking.
Beneficial transfer. Useful transfer means preserving and reusing knowledge that remains relevant across task conditions and model sizes. Shared features learned under earlier conditions can reduce the effort needed to adapt to related later conditions. When shrinking, retaining useful behavior in the smaller model can reduce the loss of predictive ability; when expanding, carrying forward the smaller model's learned representation can provide a starting point for further adaptation. These forms of transfer help the model satisfy its original learning objective while responding to the cluster's resource demands.
In the linear family, members of a group share a linear subspace, so representations learned from some members can help learning on others. In the nonlinear family, members combine the same nonlinear features using different coefficients, allowing those features to be reused. When a group returns, retained knowledge can help restore prediction performance; carrying learned weights forward during growth provides a starting point for further adaptation.
BALANCE makes these two challenges jointly evaluable: the controller must respond to external notices while maintaining the managed model's predictive performance over the long task stream.
Q4. For the two families of benchmarks presented: what is the relationship between them, and what motivations and gaps are they seeking to address?
These are two separately constructed families of prediction tasks under the same resource and model-maintenance protocol. Their settings and roles are summarized below:
| Shared linear rules | Shared nonlinear features | |
|---|---|---|
| Relatedness | Members share a linear subspace and have distinct coefficients. | Members combine shared nonlinear features with distinct coefficients. |
| Deployed learner | A bilinear predictor with a shared low-rank representation. | An ordinary task-conditioned, two-hidden-layer MLP. |
| Capacity calibration | An exact fixed-model approximation result within the bilinear family. | Local learning, transfer, revisitation and finite-budget joint-fitting diagnostics. |
| Purpose | Make the relation between local learnability and joint capacity analytically transparent. | Examine maintenance behavior with nonlinear targets and a conventional neural predictor. |
Motivation for shared linear rules: We use simple tasks that are easy to analyze. A bilinear model within the deployment cap can represent one task group, while a single fixed model of that size cannot represent all tasks exactly. This gives clear grounds for the capacity needed by one group and the overall capacity restriction, allowing the model cap to be set accordingly. With these properties specified, it is easier to interpret maintenance strategies under task and resource changes.
Motivation for shared nonlinear features: We use nonlinear tasks and an ordinary MLP to study methods during nonlinear representation learning. Local learning, transfer, revisits and joint fitting check whether the current group is learnable, whether shared knowledge can be reused, and whether the selected model faces capacity pressure across all tasks. This lets us study how resizing interacts with knowledge retention and readaptation using standard neural-network training, and provides common test conditions for different inner learning methods.
Problem addressed: When tasks keep changing and deployment capacity is reclaimed and returned externally, how should a model be resized, trained and deployed to meet resource constraints while keeping its long-term average performance high? Under the same protocol, the two families provide tests that are easy to analyze and tests with nonlinear relationships, allowing controlled comparisons between algorithms.
I appreciate the authors’ detailed answer.
I now have a better understanding of the motivation and the problem the benchmark addresses; however, it is unclear how the authors plan to incorporate these changes into the paper.
One problem that I see is that the benchmark is trying to combine two very complex problems into one (doing continual learning and adjusting the size of the weights). It is difficult to see the relationship between the two problems, which weakens the contribution of the paper. The idea is novel: I have not seen the problem of adjusting the size of the weights in a continual scenario before. However, there is still a significant gap between this version of the paper and a version that can be accepted.
I would recommend simplifying the problem and rewriting the paper more clearly. If, after this step, adding more complexity still makes sense, it can be added as an extra contribution or ablation. It would also help to make the link between the proposal and other benchmarks that may be similar, or that tackle similar problems.
I am keeping my score.
Thank you for recognizing the potential value of the formulation and benchmark. Below, we address your questions in turn.
Q1. What gaps does this work aim to fill?
Thank you for this question. We focus on the interaction between a model's continual adaptation and the capacity allocated by its platform.
Consider a model deployed on a cluster with limited resources to provide an ongoing service. Other tasks running on the cluster may require additional resources. Can the model autonomously decide to compress its deployed weights, freeing capacity for those tasks, and then adjust its size again to use the capacity returned when they finish?
The model therefore faces two challenges: (1) performing its own deployed task; and (2) adjusting the size of its weights in response to external resource-scheduling notices.
To our knowledge, this joint problem lacks a dedicated benchmark. Providing one is our primary motivation.
More specifically, a learning method may need to shrink at an externally specified time, keep serving while preparing its replacement, and relearn when capacity returns. Knowledge and resource availability therefore evolve differently: the platform can return storage immediately, while recovering predictive knowledge takes learning.
BALANCE brings external reclamation notices, effective deadlines, matched returns and timed model maintenance into a common evaluation interface. Its contribution is to make these coupled decisions comparable across algorithms under the same conditions.
This enables concrete research questions: how early should a smaller model be prepared; how much learning should precede its deployment; how often should an unchanged-size model be refreshed; and how should a policy change when maintenance becomes slower? The benchmark supplies controlled task and resource streams for examining these questions. The rule-based maintenance algorithm provides reference strategies, and the learned-control experiments examine whether decisions can be learned through imitation and reinforcement learning.
Q2. Could you provide a simple and specific explanation of the problem you are seeking to solve? Let’s leave the mathematical aspects aside for the moment.
Thank you for this question. Imagine a continually learning model running on a cluster. It keeps receiving prediction requests and learns from their feedback. Another application starts, and the cluster notifies the model that it must reduce the size of its deployed weights by a specified deadline. A smaller replacement can be trained while the current weights continue serving requests. The model must decide when to begin preparation, how much training to perform, and when to switch weights. Later, the other application finishes and capacity returns. The model can expand again, but adding parameters does not restore knowledge removed during shrinking.
Specifically, a continually learning model must meet two challenges: (1) performing its original continual-learning task; and (2) adjusting the size of its weights in response to external resource-scheduling notices.
BALANCE separates this process into a controller and the continually learning model it manages. Based on external resource-scheduling notices, the controller adjusts the managed model's weight size so that it achieves as high an average score as possible over a long horizon. BALANCE evaluates the controller's ability to manage this process over a continuing task stream.