Online Learning for Position-Aided Millimeter Wave Beam Training

Our system model is comprised of the channel model, the received signal model, and the codebook. We assume a wideband geometric channel model that is widely used in mmWave simulations [36]. We denote $N_{\mathrm{t}}$ and $N_{\mathrm{r}}$ the numbers of transmit and receive antennas, $L_{\mathrm{p}}$ the number of rays, $T$ the symbol period, $g(\cdot)$ the combined response of matched and lowpass filtering, $\boldsymbol{a}_{\mathrm{t}}$ and $\boldsymbol{a}_{\mathrm{r}}$ the normalized transmit and receive array steering vectors, $\alpha_{\ell}$ the complex channel gain, $\tau_{\ell}$ the delay, $\phi_{\ell}^{\mathrm{A}}$ and $\phi_{\ell}^{\mathrm{D}}$ the azimuth AoA and AoD, $\theta_{\ell}^{\mathrm{A}}$ and $\theta_{\ell}^{\mathrm{D}}$ the elevation AoA and AoD of the $\ell$-th path, and $(\cdot)^{*}$ the conjugate transpose operator. The channel at the delay tap $m$ is given by

We parametrize this channel using ray-tracing assuming single antennas at both the transmitter and receiver. This means $\alpha_{\ell}$ is the channel gain of a single antenna, and the factor $\sqrt{N_{\mathrm{r}}N_{\mathrm{t}}}$ is needed to express the array gain (since $\boldsymbol{a}_{\mathrm{t}}$ and $\boldsymbol{a}_{\mathrm{r}}$ are normalized).

We assume an analog beamforming architecture with one RF chain. Denote $L$ the channel length, $P_{\mathrm{t}}$ the transmit power, $s[k]$ the known training signal, $\boldsymbol{H}_{\ell}=\sqrt{N_{\mathrm{r}}N_{\mathrm{t}}}\alpha_{\ell}\boldsymbol{a}_{\mathrm{r}}(\theta_{\ell}^{\mathrm{A}},\phi_{\ell}^{\mathrm{A}})\boldsymbol{a}_{\mathrm{t}}^{*}(\theta_{\ell}^{\mathrm{D}},\phi_{\ell}^{\mathrm{D}})$ the channel matrix corresponding to the $\ell$-th path, $v_{i}[k]$ the complex Gaussian noise $\mathcal{CN}(0,\sigma_{v}^{2})$, and $r(i)$ and $t(i)$ the mappings from the beam pair index $i$ to the combiner $\boldsymbol{w}$ and beamformer $\boldsymbol{f}$ vector indices, the received signal is given by

The channel strength is defined as the squared norm of $\boldsymbol{h}_{i}=[h_{i}[0],\dots,h_{i}[L-1]]^{\mathrm{T}}$, i.e.,

The effective channel $\boldsymbol{h}_{i}$ can be estimated from noisy signals, for example using a least-squared estimator [37]. The impact of noise on the beam alignment has been studied in [1, 38], where it is shown that the impact is small for a wideband system. For a clear exposition of the learning algorithms, we assume noise-free $\gamma_{i}$ in this paper.

The beam codebook used in this paper is generated using progressive phase-shift [39]. We note that this choice is not important, and other codebooks such as the DFT codebook could be used. Uniform planar arrays (UPA) are assumed at both the transmitter and the receiver. With a UPA, each beam is defined by its azimuth $\phi$ and elevation $\theta$ main beam direction. Let $G_{\mathrm{ant}}(\cdot)$ be the antenna element radiation pattern, $\Omega_{\mathrm{y}}=kd_{\mathrm{y}}\sin(\theta)\sin(\phi)$, $\Omega_{\mathrm{x}}=kd_{\mathrm{x}}\sin(\theta)\cos(\phi)$, $k=2\pi/\lambda$ be the wave number, $\otimes$ denote the Kronecker product, $N_{\mathrm{x}}$ and $N_{\mathrm{y}}$ be the numbers of elements along the x- and y-axis, and $d_{\mathrm{x}}$ and $d_{\mathrm{y}}$ be the element spacing in the x- and y-direction, a beam pointing in $(\theta,\phi)$ direction is given by [39]

We assume $d_{\mathrm{x}}=d_{\mathrm{y}}=\lambda/2$ in this work. We assume no backplane radiation and set

We use (5) for simplicity, but we can replace it with a more sophisticated pattern like that of a patch antenna.

The beams are generated such that they are separated by their 3dB beamwidth in the azimuth and elevation. Fig. 1 shows the beams for an $8\times 8$ array. The procedure starts from the broadside direction. First, fixing the azimuth angle at $0^{\circ}$, the elevation beam direction that crosses the broadside beam at the 3dB point is determined numerically. We then do the same procedure in the azimuth while fixing the elevation angle until all $360^{\circ}$ are covered (call these the tier 1 beams). Next, we compute the elevation beam direction that crosses the tier 1 beam at the 3dB beamwidth and determine all the azimuth directions until all $360^{\circ}$ are covered. This is repeated until the main beam direction in elevation exceeds $90^{\circ}$, i.e., reaching the backplane direction.

To generate the data, we parametrize the channel model (1) using a commercial ray-tracing simulator, called Wireless InSite [40]. Ray-tracing ensures spatial consistency in the channels, which is essential to our location-based learning problems. The ray-tracing simulation is shown in Fig. 2. We assume a V2I setting in an urban street canyon. The street has two lanes, and there are two types of vehicles represented by metal boxes: cars ($1.8\mathrm{m}\times 5\mathrm{m}\times 1.5\mathrm{m}$) and trucks ($2.5\mathrm{m}\times 12\mathrm{m}\times 3.8\mathrm{m}$). The mobile user (MU) is a car on the far side lane from the BS. Because it is larger, a truck can cause blockage to the MU. We assume roof-mounted MU antenna at 1.5m and BS antenna at 7m. We generate the channel in a per snapshot basis. In each snapshot, the MU is placed uniformly at random within a location bin $[d_{0}-\sigma_{\mathrm{d}},d_{0}+\sigma_{\mathrm{d}}]$ in the far side lane, and all other vehicles are placed randomly with their gaps following an Erlang distribution. We refer to [1] for a detailed description including the material properties used for the ray-tracing. We set the carrier frequency to $60$GHz, $d_{0}=30$m, and $\sigma_{\mathrm{d}}=2.5$m, which corresponds to a location bin of $5$m length. Note that our method only requires that the position be accurate enough to identify the location bin. The system bandwidth is set to $1.76$GHz, which is used in IEEE 802.11ad. The system bandwidth is used to compute the symbol period used in the channel model (1). Using these parameters, we generate 10,000 channel samples. We assume $16\times 16$ UPA at both the MU and the BS when computing the channel strengths. The codebook described in Section II-A has 271 beams.

The idea of our proposed approach is to learn from past beam measurements experienced in a given discretized location. The measurements themselves and/or learning parameters are stored and maintained at the BS. In the offline learning case as in our prior work [1], a sufficient number of beam measurements are assumed collected before they are used to recommend beam pairs. In the online learning case considered in this paper, the learning parameters of a location bin are updated every time a beam alignment attempt is made and a new set of beam measurements becomes available at that location bin.

Assuming the database is available for making recommendation, we describe how the beam alignment procedure works. Fig. 4 illustrates the position-aided beam alignment, which consists of two phases. We start with the uplink. In Phase 1, the MU sends a training request along with its context information to the BS. In this work, we use position as the context. The BS uses the position and the database it maintains to determine a subset of promising beam directions, denoted by $\mathcal{S}$. The size of the set $\mathcal{S}$ is a system parameter chosen to balance the training overhead and the alignment accuracy. The BS then responds with an acknowledgment and the beam pair subset $\mathcal{S}$ to the MU. Since the beams are not aligned in this phase, a lower frequency control channel or mmWave with a large spreading factor can be used. In Phase 2, the beam pairs in $\mathcal{S}$ are trained and the best beam index is fed back at the end. MmWave with directional beams is used during this phase.

In the downlink, Phase 1 changes slightly. The process starts with the BS sending a training request to the MU, which then responds with an acknowledgment including its position. Next, the BS sends the list of promising beam pairs $\mathcal{S}$. The beam training in Phase 2 is kept the same. This is possible because of the reciprocity in the AoAs/AoDs, where the AoAs become the AoDs and vice versa when reversing the transmitter and receiver role. This AoA/AoD reciprocity only depends on the reciprocity property of electromagnetic waves, which holds when they propagate in passive medium like wireless channels (excluding the device’s circuits) [39].

Finally, we emphasize that this beam alignment framework is flexible and different beam pair selection methods can be used for the “Beam pair selection” block. In particular, we proposed an offline learning approach in [1] (briefly described in Section III-C) that assumes the past beam measurements are already collected. In this paper, we propose online algorithms to select the beam pairs, which will be detailed in Section IV.

There are many ways to quantify the accuracy of beam alignment. Here, we use the power loss and the power loss probability. The power loss can capture the severity of the misalignment and differentiate whether the current alignment is 3dB or 10dB away from the optimal alignment. This is important because while both cases are misaligned, the former likely still provides a good link while the latter likely cannot.

We start with the definition of the power loss. It is the ratio of the channel strength between the selected beam pair indexed by $s$ and the optimal channel strength given by

where $\mathcal{B}$ denotes the set of all possible beam pairs in the codebook. The beam pair $s$ is selected from the selection set $\mathcal{S}$, and with accurate beam training $s=\arg\max_{i\in\mathcal{S}}\gamma_{i}$. If the codebook is used without any modification, then $\mathcal{S}\subset\mathcal{B}$ and $\xi\geq 1$ always holds. The proposed online learning method also includes a component to refine the beam pairs to adapt the codebook to the environment, in which case $\xi<1$ is possible.

We quantify the beam alignment accuracy by the power loss probability defined by

for some constant $c\geq 1$. We call the case when $c=1$ the misalignment probability. A related concept to the misalignment probability is the probability of being optimal, given by

where $i^{\star}=\arg\max_{i\in\mathcal{B}}\gamma_{i}$ denotes the index of the optimal beam pair. We note that

where (10) follows because $\xi\geq 1$ (without refinement). $P_{\mathrm{opt}}(\mathcal{S})$ is used in the optimal beam selection method described in the next subsection.

In this subsection, we review two offline beam pair selection methods from [1] which form the basis for our online solution. We begin by describing the offline database. It is assumed that this database has already been collected before it is used for beam pair selection. The database has $N$ observations, where each observation consists of the channel strengths of all beam pairs combinations in $\mathcal{B}$ measured in a location bin. It is assumed that each observation is measured within a beam coherence time so that the spatial channel has negligible change [42]. An example of the database is shown in Table I. We note that some tradeoff between the performance and the cost for collecting and storing the database is possible (see [1] for details).

We now describe the two beam pair selection methods from [1], namely AvgPow and MinMisProb. AvgPow is a heuristic that selects the beam pairs by their average channel strengths. Denote $\bar{\gamma}_{i}$ the sample average of the channel strength of the $i$-th beam pair and $\underset{i\in\mathcal{B};M}{\arg\max}\{\cdot\}$ the operator that returns the top-$M$ indices, the selection set of size $|\mathcal{S}_{\mathrm{AP}}|=M$ can be written as

MinMisProb is an optimal selection method that minimizes the misalignment probability. It can be shown that this optimal selection reduces to a greedy selection using the probability of being optimal [1]. Let $|\mathcal{S}_{\mathrm{MMP}}|=M$ and denote $\hat{P}_{\mathrm{opt}}(i)$ the probability of being optimal of the beam pair $i$ estimated from the database, then

In this paper, we develop a risk-aware online learning version of MinMisProb, where beam pairs selected by the UCB indices are rejected with a probability reflecting their risk of low received powers. When rejected, the replacement pair is selected using both MinMisProb and AvgPow.

Our goal here is to develop an online version of the optimal beam pair selection method, MinMisProb. Specifically, the algorithm needs to solve the following optimization problem in an online setting:

where $B_{\mathrm{tr}}$ is the desired subset size (the training budget). In an online learning setting, $P_{\mathrm{pl}}(1,\mathcal{S})$ is not known, and it has to be estimated on the fly. To gain accurate knowledge of the beam pairs, each of them must be sampled multiple times, which means the learning can be very slow when $\mathcal{B}$ is a large set (i.e., when using large arrays with narrow beams). To remedy this problem, we propose to apply a heuristic to screen the beam pairs using a small offline database (of size $N$) to obtain a smaller set $\hat{\mathcal{B}}$ to apply the learning algorithm on. $\hat{\mathcal{B}}$ is obtained as the set of the unique beam pairs among the $NC$ entries of the first $C$ columns of Table I. In our simulations, the offline database size $N=5$ and $C=200$ seem to be good enough for this purpose.

We first propose a solution to (V-A) without risk-awareness. A subset $\mathcal{S}$ can be treated as a super-arm, and a single-play MAB algorithm can be used. Such an approach is not efficient because it treats each super-arm as independent and the number of super-arms is large due to the combinatorial nature of the number of all possible subsets.

A more efficient approach to solve (V-A) is to leverage the structure of $P_{\mathrm{pl}}(1,\mathcal{S})$ to take advantage of the dependence between the subsets. Specifically, we make use of the modularity property of the probability of being optimal [1]. We note that by the relationship in (11), the problem (V-A) is equivalent to a maximization of $P_{\mathrm{opt}}(\mathcal{S})$ with the same constraints, i.e.,

where we also replace $\mathcal{B}$ by $\hat{\mathcal{B}}$ as explained earlier. Since $P_{\mathrm{opt}}(\mathcal{S})$ is modular [1], it can be decomposed as

This property is due to the exclusive nature of the events that the $i$-th beam pair is optimal (i.e., having the highest channel strength). Recall that the probability of a union of exclusive events is the sum of the probability of each individual event [44]. The main implication of (18) is that the reward of $\mathcal{S}$ can be computed from the individual rewards of each of the beam pairs in $\mathcal{S}$. This means the optimal beam pair subset can be obtained by a greedy approach, where one beam pair is selected at a time. Observing this property, we propose to use a greedy UCB algorithm as shown in Algorithm 1, that selects the beam pairs greedily using their UCB indices. The UCB index of an arm is a high confidence bound of the expected reward, which consists of the expected reward seen so far and the uncertainty (the confidence margin) [28].

An important component of Algorithm 1 is the reward signal. Since the expected reward is the probability of being optimal, an ideal choice for the reward signal is

which takes the value 1 if the pair $i$ is best and 0 otherwise. In an actual setting, it is not known if a pair is the best in $\hat{\mathcal{B}}$ since only the beam measurements for the beam pairs in the subset $\mathcal{S}\subset\hat{\mathcal{B}}$ are available. The best guess would be the strongest pair among the beam pairs trained. Considering this limitation, we propose to use an alternative and practical reward signal,

which takes the value 1 for the pair with the strongest beam measurement in $\mathcal{S}$ and 0 for all other pairs in $\mathcal{S}$. Denoting $X_{\mathrm{tot}}[i]=\sum_{t=1}^{n}x_{i,t}$, the expected reward of beam pair $i$ at time $n$ is estimated by $\hat{P}_{\mathrm{opt}}(i)=X_{\mathrm{tot}}[i]/T_{i}$, where $T_{i}$ is the number of times that the pair $i$ was selected up to time $n$.

An intuitive understanding of this alternative reward definition can be drawn from an analogy to a sport tournament. In each round, the winners from each subgroup from the previous round play against each other to decide who will proceed to the next round, which eventually will reach the championship. We, thus, expect that over time only strong beam pairs will receive rewards of 1. We believe that under certain assumption on the underlying reward statistics of the beam pairs, it is possible to provide some guarantee that $\hat{P}_{\mathrm{opt}}(i)$ will converge to the true $P_{\mathrm{opt}}(i)$ as $T_{i}\to\infty$. This is outside the focus of this paper and is left for future work.

As will be seen in Section VII-A, Algorithm 1 does not perform well. The main reason for this is because it only tries to minimize the cumulative regret and is oblivious to the multiple-play setting in the beam alignment problem. Since multiple beam pairs are trained, the subset $\mathcal{S}$ can be divided into two parts. One part is for exploitation that uses the knowledge obtained so far to select the beam pairs and the other part is for exploration that aims at improving the accuracy of the learning parameters. By balancing these two parts, it is possible to reduce the risk (large power loss events) at any given round. In other words, in the multiple-play setting, the risk of large losses can be traded off with the speed of learning (time to get accurate statistics of the arms). Another point for improvement in Algorithm 1 is that it throws away the magnitude information because the reward signal is binary. Recall that the binary reward signal is needed because we make use of the modularity of $P_{\mathrm{opt}}(\mathcal{S})$ that allows the greedy selection using the UCB indices. To remedy these weaknesses, we propose a risk-aware version of Algorithm 1.

We first start with the definition of risk. A possible choice for the risk is the power loss, which measures the misalignment severity. Since only the beam pairs in $\mathcal{S}$ are trained, the channel strength of the optimal beam pair is not necessarily known (especially, during the early stage of the learning) and the power loss cannot be computed directly. Another important point is that this risk needs to be estimated. Therefore, it is crucial to quantify the uncertainty of the risk estimate for it to be useful for the beam pair selection. For these two reasons, we propose to use a binary risk signal defined in terms of the ratio of the channel strength of the beam pair and the best beam pair in $\mathcal{S}$, i.e., the risk signal of the beam pair $i$ at time $t$ is given by

where $\Gamma_{\mathrm{risk}}$ is a risk threshold. The choice of $\Gamma_{\mathrm{risk}}$ will be discussed in Section VII-B.

A way to capture the uncertainty is to put a prior distribution on the risk based on the observations seen so far (i.e., a Bayesian approach). By the definition (21), $z_{i,t}$ is Bernoulli distributed with some unknown parameter $\zeta$. It is well-known that the Beta distribution is the conjugate prior to the Bernoulli distribution [45]. This means that the belief on the risk of the beam pair $i$ upon seeing the measurements up to time $n$ can be updated conveniently by updating the parameters of the Beta distribution. Specifically, denoting $Z_{\mathrm{tot}}[i]=\sum_{t=1}^{n}z_{i,t}$ the prior is updated as

Here, we assume that at time 0 without any observation, $\tilde{Z}_{0}\sim\mathsf{Beta}(1,1)$, which is the uniform distribution over $[0,1]$. This is a reasonable assumption since no information on the beam pair $i$ is available at time 0.

We next explain how the risk estimate along with the prior are used in the rejection mechanism to reduce the probability of large power loss events during the learning. The new algorithm is shown in Algorithm 2, which we call the risk-aware greedy UCB algorithm. The new addition to Algorithm 1 is the risk-aware feature that rejects a beam pair selected by the greedy UCB with a probability reflecting its risk. The rejection probability is determined using the risk drawn from the prior distribution given in (22) and the confidence margin in two steps. First, a random variable $\tilde{Z}_{n}$ is drawn from this prior (line 13). Then, it is multiplied by the confidence margin for those beam pairs with $X_{\mathrm{tot}}[\ell]>0$. The obtained $Z_{n}$ is the rejection probability. The second step is needed because any beam pair is subject to blockage and their risks are not zero. This means that if $\tilde{Z}_{n}$ is used directly as the rejection probability, even good beam pairs will be rejected with non-zero probability even when $n\to\infty$. The second step ensures the algorithm accepts the UCB selection for “good” beam pairs with increasing probability over time.

The proposed rejection mechanism is a random method that rejects the beam pair with a probability $Z_{n}$. First, the algorithm draws a Bernoulli random variable $\mathsf{Rej}$ with parameter $Z_{n}$. If $\mathsf{Rej}=0$, the algorithm accepts the beam pair, otherwise it rejects the pair. In that case, the replacement beam pair is selected using $\hat{P}_{\mathrm{opt}}(i)$ when there are pairs with $X_{\mathrm{tot}}[i]>0$, and using the average channel strength $\bar{\gamma}_{i}$ when all remaining pairs have $X_{\mathrm{tot}}[i]=0$. Unlike Algorithm 1, which does not use the amplitudes of the beam measurements $\gamma_{i,n}$, here they are used to update the risk parameters and also used for the replacement selection. This new algorithm makes a fuller use of the measurement information as compared to Algorithm 1.

In this subsection, we derive regret bounds of the two algorithms that will provide insights on the effect of the rejection mechanism we introduced in Algorithm 2. We make a few simplifications to the problems to allow tractable analysis which we will describe in detail when presenting the results. Proofs are provided in Appendix A and B.

Before stating the results, we first describe the metric used for the evaluation. For this type of online learning problem, a widely used metric is the cumulative regret. It is defined as the cumulative performance loss as compared to the performance of an oracle that always plays the best arm [29]. Translating this to the beam alignment problem, the regret incurred in a time step is non-zero when the algorithm does not select the best subset of beam pairs $\mathcal{S}^{\star}$. Assuming $|\mathcal{S}^{\star}|=B_{\mathrm{tr}}$, the optimal selection in (13) tells us that $\mathcal{S}^{\star}$ contains the top-$B_{\mathrm{tr}}$ beam pairs with the highest probability of being optimal $P_{\mathrm{opt}}(\cdot)$. Now, we call the beam pairs with the $B_{\mathrm{tr}}$-highest $P_{\mathrm{opt}}(\cdot)$ as optimal and the rest of beam pairs as suboptimal. Then, the cumulative regret increases whenever one or more suboptimal beam pairs are selected in the selection set $\mathcal{S}$.

In the following, we present what is called a problem-dependent bound on cumulative regret (we drop ‘cumulative’ from now on for convenience), which quantifies the regret in terms of the optimality gap. The optimality gap is defined as the difference in the probability of being optimal for an optimal pair $i^{\star}$ and a suboptimal pair $\ell$, i.e.,

By definition, $0<\Delta_{\ell,i^{\star}}<1$ if $P_{\mathrm{opt}}(i^{\star})>0$. Note that $\Delta_{\ell,i^{\star}}$ measures the difficulty in discriminating the suboptimal pair $\ell$ from the optimal pair $i^{\star}$ for the particular problem at hand; thus, the name problem-dependent when the regret bound is expressed using optimality gaps.

For Algorithm 1, we assume the reward signal during the learning is the ideal reward and not the alternative one, i.e., we assume the reward signal is given by (19) instead of (20). We make this assumption because it is intractable to deal directly with the dynamics of the alternative reward signal in (20). The main step in deriving the regret bound is the application of the Chernoff-Hoeffding inequality to bound the probability that the sample average of the reward is within the UCB value. To apply the Chernoff-Hoeffding inequality, it is required that the sample rewards are IID, which cannot be guaranteed when using the alternative reward definition because its distribution depends on the history of the selection done so far. This, however, is a reasonable assumption, since we expect that (20) will approach (19) for large $n$, which is the domain where the regret bound is meaningful.

Theorem 1 shows a regret bound for Algorithm 1. The first term, which increases with $n$, dominates the bound. It increases as $\Delta_{\ell,i^{\star}}$ decreases. This makes sense because a small $\Delta_{\ell,i^{\star}}$ means more samples are needed to differentiate the suboptimal pair $\ell$ from the optimal pair $i^{\star}$ with confidence. The regret bound is $\mathcal{O}(\log(n))$, which is known to be optimal up to the constant coefficient in front of $\log(n)$ [29]. This confirms that the algorithm is a reasonable solution.

For Algorithm 2, we make two additional assumptions besides the accessibility of the ideal reward signal. The first assumption is that the rejection probability of any beam pair $\ell$ is constant, denoted by $1-\zeta_{\ell}$. This is used because the rejection probability of the algorithm is dynamic (depending on the observation so far) and is not tractable. With a large enough $n$, we expect the risk estimate to stabilize, and thus this is not an unreasonable assumption. The second assumption is that when rejected the replacement selection has an optimality gap $\tilde{\Delta}_{\ell,i^{\star}}$.

Theorem 2 shows a regret bound for Algorithm 2. The algorithm still has $\mathcal{O}(\log(n))$ regret but with a larger constant. This shows that introducing risk-awareness increases the learning time in the sense that $R_{2}[n]>R_{1}[n]$. This is because by rejecting a high-risk beam pair, the algorithm loses the chance to get information on that beam pair. The idea of Algorithm 2 is to distribute the learning of these high-risk beam pairs (which has high cost) more evenly among the users by rejecting them with some probability. In other words, Algorithm 2 tradeoffs the learning speed to balance the risk of severe misalignment endured by each user at different stages of the learning.

We formulate our beam pair refinement as a stochastic CAB problem. The beams are generated by progressive phase-shift and are defined by their main beam directions. The goal is to find the pointing direction of a beam pair to maximize the average channel strength of that beam pair in an online setting. Specifically, denoting $\phi^{\mathrm{t}}_{i},\theta^{\mathrm{t}}_{i},\phi^{\mathrm{r}}_{i},$ and $\theta^{\mathrm{r}}_{i}$ the transmit and receive main beam directions in the azimuth and elevation of the beam pair $i$ defined in the codebook, and $\Phi^{\mathrm{t}}_{i},\Theta^{\mathrm{t}}_{i},\Phi^{\mathrm{r}}_{i},$ and $\Theta^{\mathrm{r}}_{i}$ the corresponding 3dB beamwidths, the problem of refining the beam pair $i$ can be written as

Any pointing direction $(\phi^{\mathrm{t}},\theta^{\mathrm{t}},\phi^{\mathrm{r}},\theta^{\mathrm{r}})$ satisfying the constraints is an arm in this problem. The space to search for the best arm is the hyperrectangle defined by the constraints, which is a continuous space. This means the directions are fine-tuned within the 3dB beamwidths of the original beam pair $i$ defined by the pointing direction $(\phi^{\mathrm{t}}_{i},\theta^{\mathrm{t}}_{i},\phi^{\mathrm{r}}_{i},\theta^{\mathrm{r}}_{i})$. The coarse search to within the 3dB beamwidth is supposed to be done by the beam pair selection algorithm.

HOO is a CAB algorithm that runs on a tree. We start by describing the search tree. Then, we describe the flow of HOO. Finally, we explain the modifications made to the original algorithm to fit the beam refinement task. We describe the algorithm for refining a beam pair $i$. Since all the description is in the context of this beam pair $i$, we drop explicit references to beam pair $i$ here to avoid notational clutter.

We now define the search tree $\mathcal{T}$ which HOO runs on. Each node in the tree is a pair of transmit and receive pointing directions $(\phi^{\mathrm{t}},\theta^{\mathrm{t}},\phi^{\mathrm{r}},\theta^{\mathrm{r}})$ satisfying the constraints in (VI-A). The root of the tree is the original pointing direction of the beam pair $i$ $(\phi^{\mathrm{t}}_{i},\theta^{\mathrm{t}}_{i},\phi^{\mathrm{r}}_{i},\theta^{\mathrm{r}}_{i})$ defined in the codebook. Each node in the tree at depth $\ell<\ell_{\max}$ has 16 children which correspond to all possible combinations of transmit and receive beam directions perturbed by $1/2^{\ell}$ of the beamwidths in the four variables. Denote $(\phi^{\mathrm{t}}_{\ell,k},\theta^{\mathrm{t}}_{\ell,k},\phi^{\mathrm{r}}_{\ell,k},\theta^{\mathrm{r}}_{\ell,k})$ the parameters of the $k$-th node at depth $\ell$ in $\mathcal{T}$, its set of 16 children nodes can be written using a Cartesian product as

Using this node expansion rule, a node at depth $\ell+1$ deviates from its parent node in the pointing direction by $\mathrm{beamwidth}/2^{\ell}$ and depends only on $\ell$. Each depth in the tree can be thought of as a grid partitioning the search space defined by the constraints in (VI-A). The grid becomes finer deeper in the tree (i.e., as $\ell$ increases).

We now describe how the modified HOO works. A pseudo-code is shown in Algorithm 3. It runs on a finite tree with a maximum depth of $\ell_{\max}$. The nodes in the tree are activated on the fly, and only the root node and its children are active at time $n=0$. Thus, the initial tree is $\mathcal{T}=\{(1,1)\}\cup\mathcal{C}_{1,1}$, where $\mathcal{C}_{\ell,k}$ denotes the set of the indices of the children of the node $(\ell,k)$. In each iteration, there are three main parts. First, a node is selected by traversing the active tree starting from the root following the path through nodes that have the largest B-values (line 6-13), which is the best optimistic estimate of the average rewards of the nodes. The second part is the beam measurement for the selected node (line 14). Lastly, after obtaining the measurement, the learning parameters are updated. If the condition is met, a node in the tree is expanded, i.e., activating its 16 children nodes. Note that to lower the risk of expanding a suboptimal node, it is enforced that a node can be expanded only after it is sampled $K_{\mathrm{exd}}$ times. The last part of the parameter update is the B-values. They are computed by back calculation from the sampled node back to the root (line 33). The B-value of node $(\ell,k)$ is the minimum between its own U-value and the maximum B-value of its children nodes, i.e.,

The U-value is similar to the UCB value, but it also accounts for the smoothness property (line 23). The U-value provides an optimistic estimate of its average reward using the parameter of the node, and the maximum B-value among its children nodes provides another optimistic estimate of its reward. By taking the minimum between the two, the obtained B-value provides a refined optimistic estimate of the average reward of the node.

We introduce three main modifications to the original HOO tailored to the beam refinement setting. The first one is the use of a finite tree. The original HOO assumes an infinite tree to represent the arm space. Since small adjustments (e.g., 1/8 of the beamwidth) have a small impact on the gain, a finite tree of maximum depth $\ell_{\max}$ is used instead to save computation and storage for the learning parameters. The second one is the smoothness bound. The original HOO assumes an additive offset. Due to the multiplicative nature of the antenna gains, a multiplicative smoothness coefficient $\nu(\ell)$ as shown in line 23 is more suitable. The coefficient is computed using Lemma 1 which will be detailed in the last part of this subsection.

The third modification is the confidence margin. Because the original margin $\sqrt{2\log(n)/T_{\ell,k}}$ is too loose in our setting, we propose to use that of the norm-UCB (line 23) [28]. The margin $\sqrt{2\log(n)/T_{\ell,k}}$ is derived from the Chernoff-Hoeffding inequality, which is applicable to any distribution with the support in $[0,1]$. While normalizing the channel strengths by a large enough number will approximately guarantee that the support is within $[0,1]$, the average typically takes a value much less than 1 and the margin $\sqrt{2\log(n)/T_{\ell,k}}$ is too loose for reasonable learning horizons. The reason that the average is much smaller than 1 is the small scale fading nature of the wireless channel. Fading is due to the multipath effect and can cause the maximum instantaneous channel strength to be much larger than the average [37]. A good property of the new margin is that the sample variance is also used. Note that to enable regret analysis, the norm-UCB algorithm requires each arm be sampled at least $\lceil\alpha_{\mathrm{norm}}\log(n)\rceil$ at time $n$ with $\alpha_{\mathrm{norm}}=8$ [28]. This is enforced by setting the U-values of the nodes that need to be explored to infinity (see line 24). Note that we also introduce the condition $T_{\ell,k}<K_{\min}$, which is used to ensure that there are at least $K_{\min}$ samples of the node for computing the sample variance. This is needed when using a small $\alpha_{\mathrm{norm}}$.

We next state a lemma defining the smoothness property of the objective function in (VI-A). The lemma is useful for computing the smoothness coefficient $\nu(\ell)$.

We now explain how to determine the smoothness coefficient $\nu(\ell)$ based on Lemma 1. While we state Lemma 1 assuming an isotropic transmit antenna to avoid tedious notations, the same argument applies when we also include the transmit beam pattern. In particular, denoting $\phi^{\mathrm{r}\star},\phi^{\mathrm{t}\star}$ the optimal receive and transmit beam direction,

When steering the elevation only, we get the same relation as (35). If we assume square UPAs, the beam pattern in the azimuth and elevation will be the same. Since we only change the azimuth or elevation but not both per (31), the smoothness coefficient is given by

where $a>1$ is a correction coefficient to account for $\mathsf{Err}$ if deemed necessary. For convenience, we approximate the gain by $g(\cdot)$ the beam pattern at broadside as a function of the deviation from the broadside direction. Note that we will need $a$ only for large $\Delta\phi$. Deeper in the tree, the change in the angle is small and thus $\mathsf{Err}$ will become negligible. Also, for the sake of clear argument, we restrict $\Psi\leq\Phi/2$, but with a more elaborate choice of the coefficient of $\mathsf{Err}$ in the proof, we can allow $\Psi$ to be larger. This, however, is not a big concern in our setting because $\Delta\phi$ will be $\Phi/4$ or less and $\mathsf{Err}$ is restricted to a small value already.

This subsection evaluates the performance of the proposed risk-aware greedy UCB algorithm without the beam refinement option. There are two parameters to be decided when running Algorithm 2: the training budget $B_{\mathrm{tr}}$ and the risk threshold $\Gamma_{\mathrm{risk}}$ in (21). We note that our solution does not require that $B_{\mathrm{tr}}$ be fixed, but for simplicity, we assume that the same $B_{\mathrm{tr}}$ is used during the entire learning horizon. Fig. 6 shows the average 3dB power loss probability versus time for $B_{\mathrm{tr}}=10$ and 30 with different $\Gamma_{\mathrm{risk}}$. We can confirm from the figure that using a larger training budget $B_{\mathrm{tr}}$ leads to lower 3dB power loss probability, i.e., more accurate beam alignment. The learning seems to have two phases: the fast improvement phase in the early time steps and the slower improvement phase after that. For $B_{\mathrm{tr}}=30$ and $\Gamma_{\mathrm{risk}}=5$dB, this phase change happens at around time index 500. The slower learning phase starts when the algorithm has identified high-risk beam pairs (with some certainty) and learns those beam pairs at a slow pace due to the rejection mechanism. Regarding the risk threshold, the results show that the algorithm is not sensitive to the choice of $\Gamma_{\mathrm{risk}}$. As long as $\Gamma_{\mathrm{risk}}$ is not too large (e.g., less than 40dB), it performs well. The main reason for this behavior is due to the effect of the replacement selection method (line 19-23 in Algorithm 2) that selects beam pairs to replace those rejected; even if a good beam pair gets rejected due to risk overestimation (when using a small $\Gamma_{\mathrm{risk}}$), it will likely be picked up by the replacement selection.

Fig. 7 shows a performance comparison of the greedy UCB algorithm with and without the risk-awareness. The performance without risk-awareness is an order of magnitude worse than that with risk-aware. This might seem a bit counterintuitive because the regret bound of the risk-aware algorithm is higher. One way to understand this behavior is this. The goal of the UCB selection is to reach a state where we can ensure that a suboptimal arm is not selected with high probability (call this the optimal state). To reach the optimal state, each arm has to be sampled enough times ($T_{\ell,i^{\star}}^{(0)}$ in the derivation in Appendix A). Algorithm 1 samples the arms to reach this state fast, but it will expose early stage users to select more high-risk arms. Algorithm 2 balances the number of high-risk arms at any round by the rejection mechanism which results in a slower speed to reach the optimal state, i.e., a slower learning speed. By not exposing a user to too many high-risk arms, Algorithm 2 can ensure that the regret each user has to endure is not too large. In other words, although the cumulative regret is smaller (at a large enough time), users in early stages of Algorithm 1 have to sacrifice. Algorithm 2 distributes the regret more evenly among the users at different learning stages. We note that because of the large number of arms (400 to 600 as noted earlier), the time to reach the optimal state is large and Algorithm 1 is not practical as an online solution as shown in Fig. 7.

The last part of this subsection shows the effectiveness of our choice of the reward signal in (20). Specifically, we compare the accuracy of the beam selection using the average sample rewards ($\hat{P}_{\mathrm{opt}}(i)$) versus the more intuitive choice of average channel strength $\bar{\gamma}_{i}$. We also compare it with the case where we assume that the ideal reward defined in (19) is available to the algorithm during the learning. To evaluate this, we let the online learning run for 2000 time steps. We, then, use $\hat{P}_{\mathrm{opt}}(i)$ and $\bar{\gamma}_{i}$ estimated at time step 2000 to get two sets of beam selections and evaluate the two sets over 500 channel samples. We use $B_{\mathrm{tr}}=30$ and $\Gamma_{\mathrm{risk}}=5$dB for the online learning. Fig. 8 shows the 3dB power loss probability against the number of beam pairs trained. We can see that the beam pair selection using $\hat{P}_{\mathrm{opt}}(i)$ is more accurate than using the average channel strengths. Also, the plots show that the degradation due to the use of the proposed alternative and practical reward signal in (20) during the learning results in negligible loss. These results confirm the effectiveness of our choice of the reward signal in (20).

Before moving on to the beam refinement part, we provide some comments regarding the overhead and the requirement on the position accuracy. Regarding the accuracy of the alignment, as reported in [1], there is negligible loss in the data rate as compared to the exhaustive search when the average 3dB power loss probability is less than 1%. We can see in Fig. 6 that the average 3dB power loss falls below 2% quickly and reaches 1% within about 300 time steps with $B_{\mathrm{tr}}=30$ and $\Gamma_{\mathrm{risk}}=5$dB. The overhead per beam alignment attempt, which affects the instantaneous performance, is the same as the offline method, and the detailed study has been reported in our prior work [1]. We thus only provide a summary of the main results here for completeness.

The overhead per beam alignment attempt of the proposed method is determined by the training budget $B_{\mathrm{tr}}$. When using $16\times 16$ arrays at both the transmitter and receiver, $B_{\mathrm{tr}}=30$ was shown in [1] to provide negligible performance loss compared to the exhaustive search. This is the reason we chose $B_{\mathrm{tr}}=30$ to run our online beam pair selection algorithm so that the overhead in each beam training attempt is kept the same as in the offline approach from [1]. In [1], we compared the performance of the offline approach with two existing methods: IEEE 802.11ad and the method that uses the position only without the past beam measurements. The training overhead of our approach is less than a few percents of that of the IEEE 802.11ad method (the larger the array the smaller the percentage). Detailed comparison in the mobility context shows that our proposed beam alignment can support large arrays at high vehicular speed, while the IEEE 802.11ad struggles and its beam training time can eat up all the communication time before realignment is needed. It was also shown that using only the position results in severe performance loss when the blockage probability is not negligible (e.g., in a dense traffic). In other words, the value of past beam measurements increases with the LOS blockage probability.