acceptodds
Under review as a conference paper at ICLR 2027

ActBasis: Compact Action Parameterization for Fine-Grained Network Scheduling

Abstract

Deterministic networks require coordinated routes and precise transmission times so that periodic messages meet deadlines without link conflicts. Higher timing precision increases the number of available start times within a fixed scheduling window, expanding the range of scheduling choices. Deep reinforcement learning can learn these coupled decisions from interaction, yet directly parameterizing an action-selection weight for every time-grid position on each link makes the actor output grow with scheduling precision. This raises the question of whether fine-grained action selection requires a correspondingly large actor output dimension. We propose ActBasis, which predicts compact coefficients over a fixed temporal basis to generate action-selection weights across the full scheduling grid. Together with routing weights, these action-selection weights guide the selection of a feasible route and transmission start time. Network experiments under shared feasibility constraints show that timing-score representations support effective action selection at high scheduling precision. ActBasis retains this performance level while reducing timing output dimension by up to a factor of 125 relative to direct matrix parameterization. Controlled experiments show that compact parameterization shapes scheduling behavior even before training. The results show that compact actor outputs can support fine-grained action selection while maintaining effective scheduling performance.

Primary area:
reinforcement learning
Keywords:
deep reinforcement learning, compact action parameterization, action representation, fine-grained scheduling, network scheduling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_nv5QOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ActBasis parameterizes network-scheduling actions using compact coefficients over a fixed temporal basis, reconstructing timing scores over the full scheduling grid before selecting a feasible route and transmission time. The paper provides representation and decision-stability analyses and evaluates the method on an A380-style network and a controlled single-resource task. The results show scheduling performance close to dense timing-score policies, with up to 125× fewer timing outputs per link and a measured 9.81× reduction in replay transition payload.

Critical strengths
  1. ActBasis substantially reduces timing-output dimensionality while retaining comparable scheduling performance under a matched encoder, TD3 learner, critic interface, and decoder. At the high-resolution 1,000-message setting, it achieves 94.87% success versus Dense’s 95.30%, using 64 rather than 8,000 timing outputs per link. The measured replay-payload reduction makes this compression result practically relevant.

  2. The controlled study separates grid resolution from coefficient dimension and includes independent test instances, numerical-precision controls, and untrained-policy comparisons. These experiments show both a capacity–performance trade-off and an initialization effect, providing more information about the representation than a final-performance comparison alone.

Critical weaknesses
  1. The initialization results in Section 5.4 make this distinction consequential. At the finest grid, untrained ActBasis-64 achieves 68.94% success, compared with 69.61% after training and 66.93% for trained Dense. The paper acknowledges that much of the advantage exists before learning, but the current controls do not explain which properties of the representation produce this advantage or whether other compact parameterizations would provide similar benefits.

  2. The paper lacks comparisons with other compact action representations. Without comparisons against methods such as an interpolated low-resolution score head or another fixed basis, it is difficult to assess the advantage of the proposed harmonic parameterization over other compact alternatives.

Questions for authors
  1. How many independently generated network traffic instances underlie Table 1? Do any existing network results evaluate workloads that were not used for policy optimization or checkpoint selection?

  2. What motivates the choice of a fixed temporal basis in Section 4.1? How does the proposed parameterization differ from a compact score head followed by interpolation?

  3. Section 5.4 shows that much of the single-resource advantage is already present before training. Do the existing network runs show a similar initialization advantage, and how much additional improvement comes from learning?

3 comments

Thank you for your feedback.

Q1: How many independent traffic instances underlie Table 1? Was any network traffic evaluated that was not used for optimization or checkpoint selection?

Res1: The original Table 1 was based on a single A380 traffic sequence of 1,000 messages. Each of the six topologies in the new Table 1 uses a separately generated traffic instance. Message attributes are fixed once generated. Different grid sizes H within the same topology share the same traffic instance, and three seeds are used to compute the standard deviation across training runs.

Policy optimization and checkpoint selection in the network experiments use their corresponding fixed instances, and no separate test traffic was generated, as our goal is to compare how different action representations solve the same scheduling problem. The single-resource experiments instead use separate training, validation, and test instances, as described in Appendix B.3. Generalization of network policies to unseen traffic is the focus of our next research phase, since, as reviewer ovMJ noted, such changes may affect the feasible set, deadlines, and state distribution.

Q2: Why was a fixed temporal basis adopted, and how does this differ from interpolation following a compact scoring head?

Res2: As shown in §4.1, Figure 8, and Appendix E.1, a fixed basis uses a small number of coefficients to generate action values for the full action matrix. The harmonic basis (which we use) determines the scores at all action positions through sine and cosine components, while linear interpolation generates piecewise-linear scores from adjacent node values. Both can be viewed as fixed basis mappings.

The distinction lies in the basis structure. Linear interpolation is equivalent to a first-order piecewise-linear, locally supported basis, where the score at each position is determined solely by adjacent nodes. In contrast, the harmonic basis consists of global frequency components, with each coefficient influencing the scores at all action positions. Our newly added basis-comparison experiments on the controlled single-resource task (Appendix E) support the advantage of the global, ordered structure. As shown in Table 14, the cubic B-spline (also a locally supported basis) reaches 67.60% after training, below the harmonic basis (69.61%) and DCT-II (69.29%). This indicates that a global, ordered frequency structure preserves scheduling capability better than local interpolation, with our default harmonic basis achieving the highest overall performance.

Q3: Do the network runs show a similar initialization advantage, and how much additional improvement comes from learning?

Res3: As shown in our newly added Table 2, the network runs do not exhibit a consistent initialization advantage like that of the single-resource task, but our ActBasis maintains scheduling performance comparable to Dense both before and after training. The single-resource task removes the additional effects of routing and cross-link coordination, whereas action-value estimation in the network must jointly satisfy multi-link and periodic constraints, so the relative performance at initialization varies with the topology and grid size.

As shown in Table 13, across the 30 topology–grid pairs where multiple flows compete for limited slots, training improves the success rate by 0.38 percentage points for Dense and 0.77 for ActBasis under the same episode budget. The gains vary by topology and grid size, with the S topology at H = 400 showing the largest improvement (2.03 and 3.36 percentage points, respectively). Complete results are provided in Appendix D.3.

Thank you for the clear response. They address my questions, and I look forward to seeing what you find about how ActBasis generalizes to unseen network traffic.

Thanks for the constructive feedback. We are glad our response was helpful and will keep working on ActBasis's generalization to unseen traffic.

Reviewer_PL2NOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

To address the problem of fine-grained network scheduling, the authors propose a compact parameterization, A=FH(q)=tanh(qKHT/d)A=F_H(q)=tanh(qK^T_H/\sqrt{d} ), as an alternative to the conventional time-grid-based parameterization (A∈RR×HA\in R^{R\times H}). By combining fixed basis functions with learnable parameters, ACTBasis achieves an efficient representation with minimal modifications to the original architecture, suggesting potential applicability to a broad range of existing methods. Experimental results show that ACTBasis achieves a success rate of 94.87%, compared with 95.3% for the dense representation, while the computational overhead is theoretically significantly reduced.

Critical strengths

S1.\textbf{S1.} The paper presents an interesting perspective and compelling motivation for optimizing scheduling problems. Specifically, it explores how to represent the action space efficiently over a continuous time grid. Conventional discretization approaches can incur substantial computational overhead. An elegant mathematical solution to this problem could offer valuable insights for the broader reinforcement learning community.

S2.\textbf{S2.} The proposed solution is remarkably simple. By combining fixed basis functions with a learnable parameterization, ACTBasis requires only minor architectural modifications. This simplicity suggests that it could be readily integrated into other methods and frameworks.

Critical weaknesses

W1. The paper is overly abstract and difficult to follow.\textbf{W1. The paper is overly abstract and difficult to follow.} The problem formulation and background introduce numerous abstract terms, such as messages and links, making it difficult for readers unfamiliar with the problem to understand its practical context and significance. The authors are encouraged to introduce the A380 example from the experiments at the beginning of the paper to provide an intuitive illustration of the real-world application and motivation.

W2. The experimental evidence is insufficient to fully support the claims.\textbf{W2. The experimental evidence is insufficient to fully support the claims.} The current evaluation is limited to the A380 scenario. Furthermore, as acknowledged in Appendix B.6, the results of the equal-memory learning comparisons and experiments with alternative basis families are not provided. These additional experiments are important for establishing the effectiveness and generality of the proposed method.

W3. The practical computational benefits remain unclear.\textbf{W3. The practical computational benefits remain unclear.} As discussed in Appendix B.5, ACTBasis does not currently demonstrate substantial computational efficiency gains in practical applications. Further analysis is needed to clarify whether the reduced dimensionality translates into meaningful end-to-end computational savings.

Overall, the paper presents a highly interesting idea. However, further experiments, analyses, and more efforts are needed to better establish the effectiveness and practical value of the proposed method.

Questions for authors

Q1. Generality:\textbf{Q1. Generality:} As discussed in W2, can ACTBasis generalize to a broader range of scenarios? Does it remain effective when using alternative basis families?

Q2. Practical efficiency:\textbf{Q2. Practical efficiency:} As discussed in W3, how can the reduction in parameter dimensionality translate into tangible computational efficiency gains in practical applications?

3 comments

Thank you for your thorough review. We apologize that the original manuscript did not provide sufficient background and terminology, which obscured our motivation. We have rewritten the abstract and introduction, added Figure 1 to illustrate the effect of refining the time grid, and adjusted the related statements in the original appendix in the newly submitted manuscript.

Q1: Does ActBasis remain effective across different scenarios and basis function families?

Res1: As shown in Table 1, Figure 5, and Appendix C.1, we have added a comparison across six network topologies, each evaluated at six time-grid resolutions. ActBasis maintains scheduling performance close to Dense (the direct one-weight-per-action parameterization).

We also added experiments with different basis families on the controlled single-resource task. As shown in Figure 7 (§5.5), the ordered harmonic (default), DCT-II, and cubic B-spline bases all outperform Dense, while the random basis performs worse than Dense. As shown in Appendix E, our default harmonic basis achieves the highest overall performance.

Q2: How does reducing the output dimension translate into practical resource gains?

Res2: We agree this connection is important. However, this paper primarily investigates whether compact coefficients can replace the bulky action-value outputs while preserving scheduling capability. Reviewer ovMJ also highlighted as a main strength that our method "cleanly decouples scheduling precision from actor output dimension" while achieving performance close to Dense.

Reducing the output dimension significantly shrinks the actor's output layer. In the newly submitted manuscript, we have moved this discussion from the main text to Appendix D. As shown in Table 10, the per-transition replay payload on A380 is reduced by 9.81×.

Regarding the memory-related comparisons raised in W2, Table 10 provides the capacity comparison under a fixed replay budget, and Table 11 provides the FP16 numerical-precision control. Training-performance comparisons under equal memory budgets target training optimization for memory-constrained scenarios, which differs from the focus of this paper and is therefore beyond the scope of this paper.

Thank you for the authors’ response. The concerns raised in W1 and W2 have been largely addressed. However, the response to W3 remains insufficient, leaving me with concerns about the practical value of the proposed method. Nevertheless, considering the authors’ efforts and the clarifications provided in the rebuttal, I have decided to increase my rating to 2.

Thank you for acknowledging our additions addressing W1 and W2.

Regarding W3, we would like to further clarify the practical benefits of action representation compression. On A380 with 1,000 messages and (H=8000), ActBasis and Dense achieved success rates of 94.87% and 95.30%, respectively, while the storage footprint per replay transition was reduced to approximately 1/9.81 of Dense’s (Table 10). Since the full score matrix still needs to be reconstructed during action selection and training, the total scheduling time increased from 179.3 to 188.2 seconds, an increase of approximately 5% (Table 12). This result demonstrates the practical value of substantially reducing replay storage while maintaining comparable scheduling performance with a modest increase in runtime.

This approach also has potential applications in robot control. OpenVLA (Kim et al., 2025) uses a shared set of 256 action tokens to represent discretized values of continuous robot actions dimension by dimension, allowing action prediction through the language model’s token outputs. Our work provides a possible approach to supporting finer-grained action selection with compact representations, which could be explored for high-precision robot control in vision-language-action models.

Kim, M. J., et al. (2025). OpenVLA: An Open-Source Vision-Language-Action Model. In Proceedings of the 8th Conference on Robot Learning, PMLR 270, pp. 2679–2713.

Reviewer_ovMJOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work designs an action parameterization method for fine-grained network scheduling. Instead of producing an action score for each timing slot, the actor network outputs a fixed number of coefficients that can be converted into scores over the full scheduling period by a shared temporal basis. A deterministic decoder then selects the route and transmission start time based on the outputs of actor. By this design, the method can maintain a fixed size of actor outputs and avoid the re-training of actor when the granularity of time scheduling changes. Experiments show that the compact action representations can achieve effective action selection at high scheduling precision.

Critical strengths
  1. The proposed method cleanly decouples scheduling precision from actor output dimension, which is useful when the time grid is dynamic or becomes very fine.

  2. The method achieves scheduling performance close to dense parameterization while using much fewer actor outputs. Also, the authors have conduct controlled studies on diverse grid sizes.

Critical weaknesses
  1. The choice of basis functions is not well justified. The paper uses fixed sine/cosine harmonic bases, but does not compare them with other possible choices such as spline, polynomial, RBF, or learned bases.

  2. Generalization across scheduling windows is unclear. The experiments mainly vary the grid resolution while keeping the physical scheduling window TT fixed, and policies are optimized for individual instances. It is unclear whether a policy trained for one scheduling-window length TT can still work well when TT changes.

Questions for authors
  1. If the scheduling-window length TT changes, will the previously trained policy still work? Although the basis is defined over normalized coordinates, changing TT may also change the feasible set, deadlines, and state distribution. Does the policy need to be retrained in this case?

  2. Why are sine and cosine chosen as the temporal basis functions? Is there any task-specific motivation for using harmonic bases, and have the authors compared them with other fixed or learned basis functions?

3 comments

Thank you for your feedback.

Q1: Does changing the scheduling window T require retraining the policy?

Res1: The previously trained scheduling policy requires fine-tuning or retraining. As you pointed out, changing the scheduling window may affect the feasible set, relative time constraints, and state distribution. This paper primarily explores whether compact action representations can maintain scheduling performance across action spaces of different sizes with a fixed T. Your suggestion provides direction for future research, and we will further investigate the policy’s generalization capabilities across different values of T.

Q2: Why were sine and cosine functions chosen, and were other bases compared?

Res2: We selected sine and cosine functions based on Fourier series expansion, using a small number of coefficients to jointly generate action scores across the entire action output matrix.

To address your question, we compared DCT-II, cubic B-splines, a random orthonormal basis, and row-permuted versions of these bases. As shown in Figure 7 and Appendix E, our default harmonic basis achieved the best overall performance. We also found that disrupting the correspondence between basis values and action positions reduced performance for the harmonic, DCT-II, and B-spline bases. These results show that exploiting ordering relationships among actions helps compact representations maintain scheduling performance.

To compare basis families under the same experimental protocol and examine the role of action ordering, we evaluated only fixed bases and their row-permuted versions. The learnable basis functions you proposed represent a valuable direction for future research. We will further investigate whether learnable basis functions can adapt to different large action spaces and to variations in T.

Thanks for the responses. My concerns have been addressed.

Thank you very much for your thoughtful review and constructive feedback. We are glad that our efforts have addressed your concerns. We hope our work offers a useful perspective on decision-making in large action spaces.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.