Self-Play Ensemble Q-learning enabled Resource Allocation for Network Slicing

In this paper, a network-slicing scenario is considered for supporting two distinct types of slices, namely eMBB and URLLC slices [17]. As illustrated in Fig. 1, a two-step resource allocation scheme is employed on a gNodeB (eNB). The eNB is equipped with a mobile edge computing (MEC) server which provides the capability of offloading tasks to a cloud server. During the inter-slice phase, the eNB intelligently distributes radio resources among two slices. Subsequently, these allocated resources are utilized within each specific slice during the intra-slice phase. The primary objective is to assign radio resources which refers to time-frequency resource blocks between two slices to meet the latency and throughput requirements of URLLC and eMBB slices.

For meeting the requirements of slices, decreasing the latency is pivotal which is evaluated by the equation below:

where $d^{Tx}$ is the transmission delay, $d^{rTx}$ is the re-transmission delay, $d^{que}$ is the queuing delay, and $d^{edge}$ is the processing delay in the MEC server. $d^{Tx}$ is affected by the size of packets sent by a user equipment (UE) and the link capacity between UE $u$ and its connected eNB $j$, denoted as $C_{j,u}$. The value of $C_{j,u}$ is calculated using the equation below:

where $N_{u}$ is the set of allocated resource blocks to the UE $u$, $b_{RB}$ is a RB’s bandwidth, $N_{0}$ is noise power density, $p_{j,r}$ is the transmission power of RB $r$ in the eNB $j$. $x_{j,u,r}$ indicates a binary variable that illustrates whether RB $r$ is assigned to the UE $u$ or not, $q_{j,u,r}$ refers to the channel gain between eNB $j$ and UE $u$ over resource block $r$, $J_{-j}$ indicates the set of eNBs, except the $j^{th}$, $u_{j^{\prime}}$ is the set of UEs in the eNB $j^{\prime}$, and $N_{j^{\prime}}$ is the set of total resource blocks in eNB $j^{\prime}$. It is noteworthy to mention that computation decisions and task management functionalities can be deployed in the eNB or the non-real-time RAN Intelligent Controller from the O-RAN architecture. The readers are referred to [18] for details of O-RAN architecture. To meet UEs requirements, the URLLC slice seeks to reduce latency, while the eMBB slice aims to increase throughput. Given these considerations, the allocation of resource blocks could be formulated as follows:

where $w^{eMBB}$ and $w^{URLLC}$ are the weighting factors of eMBB and URLLC slices to make throughput and latency comparable. $B^{eMBB}_{j,u}$ represent the throughput of the UE $u$ in the eMBB slice of eNB $j$. $D^{tar}$ refers to the target delay of the URLLC slice, and $D^{URLLC}_{j,v}$ identifies the UE latency within the URLLC slice of eNB $j$. $M^{eMBB}_{j}$ and $M^{URLLC}_{j}$ are indicators of how many eMBB and URLLC slices are available in eNB $j$. Using eq. (3a) ensures that each RB is allocated to a single UE. According to eq. (3b) and eq. (3c), the total number of eNBs and computation capacity allocated to the eNB should not be greater than the number of resources available within the eNB.

The ensemble Q-learning approach involves maintaining several Q-tables, denoted as $Q_{i}$, where $i$ represents the number of tables. In the considered approach, decisions from multiple Q-tables are aggregated using a majority voting (MV) mechanism. Each agent provides its Q-values for a specific state-action pair, and the action with the majority of votes is selected as the final decision. $Q_{i}(s_{t},a_{t})$ represent the Q-value predicted by the $i$-th Q-learning agent for state $s_{t}$ and action $a_{t}$. The MV function can be defined as Algorithm 1 where $N$ is the number of Q-tables which is equal to 3, and ${a^{*}}$ is the selected action based on MV. While increasing the number of Q-tables improves the method’s exploration capabilities, it also increases the method’s complexity and energy consumption. Therefore, we select $N=3$ to balance these factors.

When implementing MV for Q-learning, it’s crucial to address ties when there is no clear majority. A random selection strategy is employed for this purpose. It must be noted that the weights of all considered $Q_{i}$ are assumed to be the same, then the system is affected by all of them equally.

Ensemble Q-learning, while effective in some scenarios, faces several challenges that can hinder its performance in dynamic network slicing environments. One of the challenges is its exploration, which increases the probability of convergence to local optima. To address this, self-play ensemble Q-learning considers its previous values, always striving to select the best action encountered thus far and avoiding getting stuck in local optima. Additionally, handling complex state spaces poses a difficulty for ensemble Q-learning. Self-play ensemble Q-learning introduces mechanisms to better understand and learn from all Q-values by the explicit consideration of the agent’s interactions with its past versions. Furthermore, the Q-values are updated not only based on the immediate rewards from the environment but also the outcomes of self-play experiences against its past versions. The update involves a weighted average of the current Q-values and the past versions of Q-values, which are calculated by the equation below.

where the weight $\beta$ determines how much influence the past version has on the current Q-values. This ensures a balance between incorporating new experiences from the current episode and leveraging knowledge from the past version. Using this equation leads to observing the performance of the current agent’s observations from the environment against its previous strategies and controlling the influence of the past versions on the updating of current Q-values. This part of the algorithm focuses on enhancing the current agent’s Q-values by considering its past strategies, allowing it to adapt and improve over time through self-play. For each $(s_{t+1},a_{t+1})$, iterate through all $(s_{t},a_{t})$ in the Q-table.

The self-play ensemble Q-learning method can improve the learning process in the following ways:

• •

  Monitoring: In the ensemble part, the agent attempts to consider various aspects of the system, choosing different actions in each state to receive diverse rewards. This process helps the agent generate diverse experiences.

• •

  Adaptability: Self-play enables the agent to adapt its strategy over time. As it encounters different situations, it learns to respond to a variety of states, making it more robust and adaptable.

• •

  Continuous Improvement: The agent improves its policies and strategies by repeatedly playing against its current or past versions, refining its decision-making process, discovering new tactics, and enhancing performance.

• •

  Reduced Dependency on External Data: Since the agent generates its training data by self-play, it becomes less dependent on external data. This is useful when external data is poisoned, limited, or not readily available.

• •

  Exploration of Strategies: During self-play, the agent explores various strategies, enhancing its understanding of the environment and potentially discovering optimal or near-optimal policies.

This process allows the current agent to learn from its recent experiences in the network slicing system, and also its history of playing against earlier versions. It introduces a form of memory or hindsight learning, potentially improving the agent’s strategy over time. Algorithm 2 provides information about the self-play ensemble Q-learning on the RL algorithm.

In resource allocation for network slicing using RL algorithms, one of the Q-tables, denoted as $Q_{i}$, may be manipulated by a malicious user. The malicious user strategically alters the Q-values to introduce a deliberate bias in the learning process. The intentional manipulation introduced by the malicious user is denoted by the equation below:

in which malicious influence disrupts the integrity of the learning process. The algorithm, which aims to converge towards optimal policies based on accurate Q-value estimates, is misled by the distorted information provided by the adversarial agent. This adversarial interference can lead to suboptimal resource allocation decisions as the algorithm’s learning dynamics are compromised. The impact is significant in dynamic environments where the learning agent relies on accurate Q-value estimations to adapt to changing network conditions.

In this paper, we considered two baselines, double Q-learning, and Q-learning, to evaluate the performance of the self-play ensemble Q-learning methods.

Considered scenario has three independent eNBs, with a cell radius of 125 meters and a bandwidth of 20 MHz, supporting 13 resource block groups. The network environment is based on the 3GPP Urban Macro network model. The eNBs operate with a PHY configuration that includes a 15 kHz subcarrier spacing, 12 subcarriers per resource block, and a transmission power of up to 40 dBm. The antenna gain is set at 15 dB, and the system operates at a carrier frequency of 30 GHz.

Each eNB includes URLLC and eMBB slices with 10 UEs and 5 UEs, respectively. Packet sizes are 50 bytes for URLLC and 100 bytes for eMBB, with traffic generated according to a Poisson distribution. The TTI size is 2 OFDM symbols, equating to 0.1429 ms. Hybrid automatic repeat request (HARQ) processes are asynchronous, featuring a round trip delay of 4 TTIs, 6 HARQ processes, and a maximum of 1 HARQ re-transmission.

Propagation characteristics include a path loss model defined as $128.1+37.6\log_{10}(D[\text{Km}])$, Log-Normal Shadowing with an 8 dB standard deviation, a noise figure of 5 dB, and a penetration loss of 5 dB. In terms of learning algorithm parameters, $\alpha$, $\beta$, $\gamma$, and $\epsilon$ are set to 0.5, 0.5, 0.2, and 0.3, respectively. The specific learning rates for the self-play ensemble Q-learning, denoted as $\alpha_{a}$, $\alpha_{b}$, and $\alpha_{c}$, are 0.7, 0.8, and 0.9, respectively.

We evaluate the performance of three algorithms, namely Q-learning, double Q-learning, and self-play ensemble Q-learning in meeting the requirements of URLLC and eMBB slices. We then evaluate the robustness of self-play ensemble Q-learning and double Q-learning methods when one of the Q-tables is affected by a malicious user . Fig. 2 illustrates the convergence speed of the algorithms. The figure highlights that self-play ensemble Q-learning exhibits higher rewards compared to the other two algorithms. Moreover, this method achieves convergence to optimal rewards quicker than alternative approaches due to benefiting from multiple Q-value ensemble learning and competition with its past knowledge in self-play.

Fig. 5 presents the latency of the URLLC slice when using Q-learning, double Q-learning, and self-play ensemble Q-learning. According to the results, the system delay with Q-learning is more significant than other algorithms, while self-play ensemble Q-learning exhibits the lowest delay. This method demonstrates 21.92% improvement in latency. Similarly, as depicted in Fig. 5, self-play ensemble Q-learning exhibits a substantial enhancement, achieving 24.22% improvement in throughput for the eMBB slice. Finally, Fig. 5 showcases notable progress, boasting 23.63% improvement in the packet drop rate (PDR) of self-play ensemble Q-learning is considerably lower than Q-learning and double Q-learning.

In the context of a scenario where one of the tables, is affected by a malicious user, as depicted in Fig. 5 to Fig. 5, the adversarial agent intentionally manipulates Q-values to mislead the resource allocation decisions of the learning algorithm. Such manipulation can significantly impact the overall performance of the system. It must be noted that double Q-learning relies on two separate Q-tables to estimate action values, the malicious agent’s influence directly skews the learning process and can misguide the algorithm’s decision-making. Then the results of resource allocation become suboptimal and deviate from the system’s desired objectives. In self-play ensemble Q-learning, degradation of the system performance occurs when the affected Q-value misleads the system with intentionally biased information. Self-play ensemble Q-learning is less affected by malicious users than double Q-learning because it uses multiple Q-tables to cross-check and detect irregular values, helping the agent avoid actions influenced by malicious behavior. This approach improves robustness by leveraging historical Q-values to identify and correct deviations, ensuring more stable learning and decision-making processes in the presence of adversarial interference.