SAI
a Sensible Artificial Intelligence that plays Go

The winrate $\rho$ of the current player depends on the state $s$. For the sake of generality we include a second parameter, i.e. a number $x\in\mathbb{Z}$ of virtual bonus points for the current player. So we will have $\rho=\rho(s,x)=\rho_{s}(x)$, with the latter being our standard notation. When trying to win by some amount of points $n$, the agent may let $x=-n$ to ponder its chances.

Since $\rho_{s}(x)$ as a function of $x$ must be increasing and map the real line onto $[0,1]$, a family of sigmoid functions is a natural choice:

Here we set

The number $\bar{k}_{s}$ is the signed komi, i.e. if the real komi of the game is $k$, we set $\bar{k}_{s}=k$ if at $s$ the current player is white and $\bar{k}_{s}=-k$ if it is black.

The number $\alpha=\alpha_{s}$ is a shift parameter: since $\sigma(-\alpha,\alpha,\beta)=1/2$, it represents the expected difference of points on the board from the perspective of the current player. The number $\beta=\beta_{s}$ is a scale parameter: the higher it is, the steeper is the sigmoid, generally meaning that the result is set. The highest meaningful value of $\beta$ is of the order of 10, since at the end of the game, when the score on the board is set, $\rho$ must go from about 0 to about 1 by increasing its argument by one single point. The lowest meaningful value of $\beta$ for the full 19$\times$19 board is of the order of $10/2/361\approx 0.01$, since at the start of the game, even for a very weak agent it would be impossible to lose with a 361.5 points komi in favor.

AlphaGo, AlphaGo Zero, AlphaZero and Leela Zero all share the same core structure, with neural networks that for every state $s$ provide

• •

  a probability distribution over the possible moves $p_{s}$ (the policy), trained as to choose the most promising moves for searching the tree of subsequent positions;

• •

  a real number $v_{s}$ (the value), trained to estimate the probability of winning for the current player.

We propose a modification of Leela Zero neural network that for every state $s$ gives the usual policy $p_{s}$, and the two parameters $\alpha_{s}$ and $\beta_{s}$ described above instead of $v_{s}$.

Training of Go neural networks with multiple komi evaluation is a challenge on its own. Supervised approach appears unfeasible, since large databases of games have typically standard komi values of 6.5, 7.5 or so and moreover it’s not possible to estimate final territory reliably for them. Unsupervised learning asks for the creation of millions of games even when the komi value is fixed. If that had to be made variable, then theoretically millions of games would be needed for each komi value¹¹ 1 The argument that one can play the games to the end and then score under multiple komi does not work here because this doesn’t allow to estimate the $\beta$ parameter. Moreover that approach would rely on the agent of the self-plays to converge to score-perfect play, while the current approach is satisfied with convergence to winning-perfect play..

Moreover, games started with komi very different from the natural values may well be weird, wrong and useless for training, unless one is able to provide agents with different strength. Finally, we are trying to train two parameters $\alpha_{s}$ and $\beta_{s}$ from a single output, i.e. the game outcome. To this aim, it would be advisable to have at least two finished games, with different komi, for many training states $s$.

We propose a solution to this problem, by dropping the usual choice that self-play games for training always start from the initial empty board position. The proposed procedure is the following.

1. 1.

   Start a game from the empty board with random komi close to the natural one.

2. 2.

   For each state in the game, take note of the estimated value of $\alpha$.

3. 3.

   After the game is finished, look for states $s$ in which $d:=|\bar{k}_{s}+\alpha_{s}|$ is large: these are positions in which one of the sides was estimated to be ahead of $d$ points.

4. 4.

   With some probability start a new game from states $s_{*}$ with the komi corrected by $d$ points, in such a way that the new game starts with even chances of winning, but with a komi very different from the natural one.

5. 5.

   Iterate from the start.

With this approach games branch when they become uneven, generating fragments of games with natural situations in which a large komi may be given without compromising the style of game. Moreover, the starting fuseki positions, that, with the typical naive approach, are greatly over-represented in the training data, are in this way much less frequent. Finally, not all but many training states are in fact branching points for which there exists two games with different komi, yielding easier training.

We incorporated in our agents the following smart choices of Leela Zero:

• •

  the evaluation of the winrate of an intermediate state $s$ is the average of the value $v$ over the subtree of states rooted at $s$, instead of the typical minimax that is expected in these situations;

• •

  the final selection of the move to play is done, at the root of the MCTS tree, by maximizing the number of playouts instead of the winrate.

However, we designed our agents to be able to win by large score differences. To this aim, we designed a parametric family of value functions $\nu=\nu_{\lambda}(s)$, $\lambda\in[0,1]$, as the average of $\sigma(x,\alpha,\beta)$ for $x$ ranging from $\bar{k}$ to a level of bonus/malus points $\bar{x}_{\lambda}$ that would make the game closer to be even: in other words, for $\lambda>0$, $\nu_{\lambda}(s)$ under- or over-estimates the probability of victory, according to whether the player is winning or losing.

Scaling the Go board from size $n$ to size $\rho n$ with $\rho<1$ yields several advantages:

• •

  Average number of legal moves at each position scales by $\rho^{2}$.

• •

  Average length of a game scales by $\rho^{2}$.

• •

  The number of visits in the UC tree that would result in a similar understanding of the total game, scales at an unclear rate, nevertheless one may naively infer from the above two, that it may scale by about $\rho^{4}$.

• •

  The number of resconv layers in the ANN tower scales by $\rho$.

• •

  The fully connected layers in the ANN are also much smaller, even if it is more complicated to estimate the speed contribution.

All in all it is reasonable that the total speed improvement for self-play games is of the order of $\rho^{9}$ at least.

Since the expected time to train 19$\times$19 Go on reasonable hardware has been estimated to be in the order of several hundred years, we anticipated that for 7$\times$7 Go this time should be in the order of weeks. In fact, with a small cluster of 3 personal computers with average GPUs we were able to complete most runs of training in less than a week each. We always used networks with 3 residual convolutional layers of 128 filters, the other details being the same as Leela Zero. The number of visits corresponding to the standard value of 3200 used on the regular Go board would scale to about 60 for 7$\times$7. We initially experimented with 40, 100 and 250 visits and then went with the latter, which we found to be much better. The Dirichlet noise $\alpha$ parameter has to be scaled with the size of the board, according to ² and we did so, testing with the (nonscaled) values of $0.02$, $0.03$ and $0.045$. The number of games on which the training is performed was assumed to be quite smaller that the standard 250k window used at size 19, and after some experimenting we observed that values between 8k and 60k generally give good results.

As explained in Section II-B, Leela Zero’s neural network provides for each position two outputs: policy and winrate. SAI’s neural network should provide for each position three outputs: the policy as before and the two parameters $\alpha$ and $\beta$ of a sigmoid function which would allow to estimate the winrate for different komi values with a single computation of the net. It is unclear whether the komi itself should be provided as an input of the neural network: it may help the policy adapt to the situation, but could also make the other two parameters unreliable²² 2 As will be explained soon, the training is done at the level of winrate, so in principle, knowing the komi, the net could train $\alpha$ and $\beta$ to any of the infinite pairs that, with that komi, give the right winrate.. For the initial experiments the komi will not be provided as an input to the net.

With the above premises, the first structure we propose for the network is very similar to Leela Zero’s one, with the value head substituted by two identical copies of itself devoted to the parameters $\alpha$ and $\beta^{*}$. The latter is then mapped to $\beta$ by equation $\beta_{s}=c\exp(\beta^{*}_{s})$. The exponential transform imposes the natural condition that $\beta$ is always positive. The constant $c$ is clearly redundant when the net is fully trained, but the first numerical experiments show that it may be useful to tune the training process at the very beginning, when the net weights are almost random, because otherwise $\beta$ would be close to 1, which is much too large for random play, yielding training problems. The two outputs were trained with the usual $l^{2}$ loss function but with the value $v_{s}$ substituted with $\rho_{s}(0)=\sigma(\bar{k}_{s},\alpha_{s},\beta_{s})$.

We used two structures of network, type V and type Y, which are described in detail in ¹⁴.

To train the network we included the komi value into the training data used by SAI. The training is then performed the same way as for Leela Zero, with the loss function given by the sum of regularization term, cross entropy for the policy and $l^{2}$ norm for the winning rate.

The winning rate is computed with the sigmoid function given by equations (1) and (2), in particular we set $v(s)=\rho_{s}(0)$ and backpropagate gradients through these functions.

To train the neural network it is clearly necessary to have different komi values in the data set. It would be best to have very different komi values, but when the agent starts playing well enough, only few values around the correct komi³³ 3 The correct komi for 7$\times$7 Go is known to be 9, in that with that value both players can obtain a draw. Since we didn’t want to deal with draws, for 7$\times$7 Leela Zero we chose a $9.5$ komi, thus giving victory to white in case of a perfect play. In fact we noticed that with a komi of $7.5$ or $8.5$ (equivalent by chinese scoring) the final level of play of the agents didn’t seem to be as subtle as it appears to be for the $9.5$ komi. make the games meaningful.

To adapt the komi values range to the ability of the current network, when the server assign a self-play match to a client, it chooses a komi value randomly generated with distribution given by the sigmoid itself. Formally,

where $\rho_{s}(x)=\sigma(x,\alpha_{s},\beta_{s})$, $s$ is the initial empty board state, $\alpha_{s}$ and $\beta_{s}$ are the computed values with current network and $U\sim\unif(0,1)$, thus giving to $K$ an approximate logistic distribution.

As the learning goes on, we expect $\alpha_{s}$ to converge to the correct value of 9, and $\beta_{s}$ to increase, narrowing the range of generated komi values.

To deal with this problem we implemented the possibility for the server to assign self-play games starting from any intermediate position.

After a standard game is finished, the server looks to each of the game’s positions and from each one may branch a new game (independently and with small probability). The branched game starts at that position with a komi value that is considered even by the network. Formally,

where $s$ is the branching position and $\pm\alpha_{s}$ is the value of $\alpha$ at position $s$, as computed by the current network, with the sign changed if the current player was white.

The branched game is then played until it finishes and then all its positions starting from $s$ are stored in the training data, with komi $k^{\prime}$ and the correct information on the winner of the branch.

This procedure should produce branches of positions with unbalanced situations and values for the komi that are natural to the situation but nevertheless range on a wide interval of values.

When SAI plays, it can estimate the winrate for all values of the komi with a single computation of the neural network. In fact, getting $\alpha$ and $\beta$ it knows the sigmoid function that gives the probability of winning with different values of the komi for the current position.

We propose the generalization of the original agent of Leela Zero as introduced in Section II-D. Here we give further details.

The agent behaviour is parametrized by a real number $\lambda$ which will be usually chosen in $[0,1]$.

To describe rigorously the agent, we need to introduce some more mathematical notation.

To provide a benchmark for the developement of SAI, we adapted Leela Zero to 7$\times$7 Go board and performed several runs of training from purely random play to a level at which further improvement wasn’t expected. More details on this step can be found in ¹⁴. A sample of 7$\times$7 Leela Zero nets formed the panel used in the evaluation phase of the SAI runs.

When doing experiments with training runs of Leela Zero, we produce many networks, which had to be tested to measure their playing strength, so that we can assess the performance and efficiency of each run.

The simple usual way to do so is to estimate an Elo/GOR score for each network⁴⁴ 4 In fact the neural network, is just one of many components of the playing software, which depends also on several other important choices, such as the number of visits, fpu policies and all the other parameters. Rigorously the strength should be defined for the playing agent (each software implementation of Leela Zero), but to ease the language and the exposition, we will speak of the strength of the network, meaning that the other parameters were fixed at some value for all matches.. The idea which defines this number is that if $s_{1}$ and $s_{2}$ are the scores of two nets, then the probability that the first one wins against the second one in a single match is

so that $s_{1}-s_{2}$ is, apart from a scaling coefficient $c$ (traditionally set to 400), the log-odds-ratio of winning.

This model is so simple that is actually unsuitable to deal with the complexity of Go and Go playing ability. In fact in several runs of Leela Zero 7$\times$7 we observed that each training phase would produce at least one network which solidly won over the previous best, and was thus promoted to new best. This process would continue forever, or at least as long as we dared keep the run going, even if from some point on, the observed playing style was not evolving anymore. When some match was tried between non-consecutive networks, we saw that the strength inequality was not transitive, in that it was easy to find cycles of 3 or more networks that regularly beat each other in a directed circle. Even with very strong margins.

We even tried to measure the playing strength in a more refined way, by performing round-robin tournaments between nets and then estimating Elo score by maximum likelihood methods. This is much heavier to perform and still showed poor improvement in predicting match outcomes.

It must be noted that this appears to be an interesting research problem in its own. The availability of many artificial playing agents with different styles, strengths and weaknesses will open new possibilities in collecting data and experimenting in this field.

A different approach which is both robust and refined and is easy to generalize is to use a panel of networks to evaluate the strength of each new candidate.

We chose 15 networks of different strength from the first 5 runs of Leela Zero 7$\times$7. Each network to be evaluated is opposed to each of these in a 100 games match. The result is then a vector of 15 sample winning rates, which contains useful multivariate information on the playing style, strengths and weaknesses of the tested net.

To summarize this information in one rough scalar score number, we used principal component analysis (PCA). We performed covariance PCA once for all the match results of the first few hundreds of good networks, determined the principal factor and used its components as weights⁵⁵ 5 Due to the asymmetric choice of the default komi at 9.5, which is favorable to white, the experiment showed that after a run reached almost perfect play as white, its playing style as black would typically get randomly worse, as the learning frantically tried to find any way for black to trick white and get an impossible win. Hence we had to use only the results of the nets playing as white to evaluate performaces. We recall that, by the properties of PCA, the principal factor will find the maximum distinguisher between results against the panel networks. In the 15-dimensions space of results this will be the direction in which the networks analysed are more different from one another. The coefficients of this factor resulted to be all positive numbers ranging from 0.001 for very weak networks, to 0.62 for the strongest one, thus confirming that these weights represent a reasonable measure of strength.. Hence the score of a network is the principal component of its PCA decomposition. This value, which we call panel evaluation, correlates well with the maximum likelihood estimation of Elo by round-robin matches, but is much easier and quicker to compute.