Transferring Deep Reinforcement Learning with Adversarial Objective and Augmentation

Reinforcement learning is a method that can learn how to map observations to actions by maximizing the reward [9]. Unlike supervised learning paradigm usually requires correct labels in order to learn a new task, the reinforcement learner is not told the best action, but instead acting according to the best reward. The chosen action affects the immediate reward, the next observation, and all subsequent rewards.

In recent years, with the significant advances of deep learning, especially with convolutional neural network, numerous researchers have attempted to use deep learning to solve the reinforcement learning tasks [1, 5, 6, 10]. Mnih et al. [1] presented Deep Q-Network (DQN) that uses a deep network to approximate the state-action value function used in traditional Q-learning [11]. It solves the storing space issue in traditional state-action tabular representation (Q-table) [11]. Van Hasselt et al. [6] further extended the DQN by solved an overestimate action values issue suffered for the DQN algorithm and leads to much better performance. Mnih et al. [5] introduced a deep reinforcement learning framework that uses asynchronous gradient descent for optimizing deep neural network controllers, that achieve state-of-the-art performance. Fortunato et al. introduced a method called NoisyNet [12], by added parametric noise to its weight to induce stochasticity of the agent’s policy, that can be used to aid efficient exploration. Replacing the conventional exploration heuristics for A3C, DQN with NoisyNet can yield higher scores for a wide range of Atari games.

Deep neural networks are able to learn powerful representations from large quantities of labeled input data, however, they usually fail to generalize to new datasets with even minor changes. In recent years, numerous domain adaption methods have been proposed [13, 14, 15, 16, 17]. Instead of collecting labeled data and training a new classifier for every possible scenario, unsupervised domain adaptation methods are trying to compensate for the degradation in performance by transferring knowledge from labeled source domains to unlabelled target domains. Sun et al. [16] proposed CORAL, that can align the second-order statistics of the source and target distributions of features with a linear transformation. Sun and Saenko [14] proposed a differentiable CORAL loss that can be minimized end-to-end and be integrated with general deep neural networks. Ganin and Lempitsky [15] used a domain classifier with a gradient reversal layer that multiplies the gradient by a negative constant, which can ensure that the feature distributions over the two domains are made similar, thus resulting in the domain-invariant features. Tzeng et al. [13] present a method that uses the concept of adversarial learning [18]; they pre-train a source encoder CNN with labeled data, then perform adversarial adaptation by learning a target encoder CNN such in order to fool the discriminator. This results in a target encoder CNN that can produce features that similar to source features.

Although an agent trained by DRL can outperform human-expert level across many Atari games, but fail to generalize across multiple gameseach agent can only play a single game. To address this issue, some researchers attempt to train a single agent to handle multiple tasks by integrate model compression technique to deep reinforcement learning. Parisotto et al. [8] first presented “Actor-Mimic”, exploited the use of DRL and model compression techniques to train a single policy network that learns how to act in a set of distinct tasks by using the guidance of several expert teachers. Yin and Pan [19] proposed a new multi-task policy distillation architecture by concatenating a set of task-specific convolutional layers and shared multi-task fully connected layers. The shared multi-task fully connected layers enable the agent to learn a generalized policy. Sharma and Ravindran [20] proposed a multi-task learning framework which solves multiple tasks in an online learning setup without the need of training multiple task-specific teacher (expert) networks.

Reinforcement learning involves an agent, a set of states $S$, and a set of actions $A$. By performing an action $a\in A$, the agent transitions from state $(s)$ to another state $({s}^{\prime})$. After performing an action, the agent will get a reward. The goal of reinforcement learning agent is to maximize its total reward.

Q-learning was first introduced by Watkin et al. [11], a reinforcement learning method that can estimate the optimal action value by interacting with the environment. The Q stands for the quality of a “state-action pair”, by interacting with environment and record all the state-action transitions, it can gradually update the Q-value of the state-action pair. The quality of a state-action pair is defined as:

For each step $t$, the agent selects action $a_{t}$ and transits from state $s_{t}$ to state $s_{t+1}$. The new Q-value is updated as the last step Q-value $Q(s_{t-1},a_{t-1})$ plus the current reward $r_{t}$ and estimate future optimal value multiply by discount factor $r_{t}+\gamma\max_{a}Q(s_{t+1},a)$, and the update equation then can describe as follow:

where $\alpha$ is learning rate and $\gamma\in[0,1]$ is a discount factor that trades off the importance of immediate and future rewards.

Although Q-learning can be used to estimate optimal policy, instead of learning all action values in all states, which may be too large to learn in complex environments, we can learn a parameterized value function $Q(s,a;\theta)$, where $\theta$ are the parameters of the network. For updating the network parameters after taking action $a_{t}$ in state $s_{t}$ and observing the reward $r_{t+1}$ and next state $r_{t+1}$ is:

and $Y_{t}$ is defined as:

where $\alpha$ is learning rate and $\gamma\in[0,1]$ is a discount factor that trades off the importance of immediate and later rewards.

A most famous deep reinforcement learning algorithm is deep Q network (DQN) that introduced by Mnih et al. [1]. DQN is a multi-layered neural network that for a given state outputs a vector of action values. For an $n$-dimensional state space and an action space containing $m$ actions, the neural network is a function from $\mathbb{R}^{n}$ to $\mathbb{R}^{m}$.

There have two important components that used in DQN, a target network and replaying memory. The target network is same as the online network, and the parameters of target network $\theta^{{}^{\prime}}$ will copy from the online network every $t$ steps, and kept fixed on all other steps. The target network is used to predict the true Q value when learning ($Y_{t}$). For the replaying memory, observed transitions are stored for some time and sampled uniformly from this memory buffer to update the network. Both the target network and the replaying memory improve the training stability and overall performance of the algorithm [1]. And to explore environment more stable and faster, DQN also using $\epsilon$-greedy[9] agent’s policy, that agent will select a random action with certain possibilities.

In this work, we split the agent into two networks as shown in Figure 1 (a), a feature encoder $F$ and a Q-value predictor $V$. Feature encoder extract features based on the input observations, and the Q value predictor uses these encoded features to predict the corresponding action Q values.

We present a framework for unsupervised domain adaption between deep reinforcement learning tasks. Assume a source environment $Env_{s}$ and a target environment $Env_{t}$, there are domain shifts between $Env_{s}$ and $Env_{t}$. We have access to the reward and next state after performed an action on $Env_{s}$.

The overview is shown in Figure 1, we first pre-trained the source task agent on $Env_{S}$, training a feature encoder $F_{S}$ and Q-value predictor $V_{S}$ that can return a vector of Q values for all possible actions by given features of observation, thus can select the best action $a$. And our goal is to learn a target feature encoder $F_{T}$ and Q-value predictor $V_{T}$ that can handle $Env_{T}$ as fast as possible.

We integrate generative adversarial network [18] concept into transfer progress, as shown in Figure 1 (b). The target feature encoder $F_{T}$ plays the role of generator, and a domain classifier $D$ that can predict the domain label (source or target domain) by seeing the encoded features output by $F_{S}$ and $F_{T}$. We then perform adversarial adaption by training the $F_{T}$ in order to to prevent the domain classifier from predicting the correct domain label from the encoder features. The target feature encoder (Generator) and domain classifier (Discriminator) are playing counterparts, and at the end of training, the target feature encoder $F_{T}$ will learn the feature representations that can be generalized across both source and target domain.

In the adversarial objective approach, the main goal is to regularize the learning of the source and target mappings, we can minimize the distance between $F_{S}(S_{s})$ and $F_{T}(S_{t})$ distributions, where $S_{s}$ and $S_{t}$ is the states of $Env_{S}$ and $Env_{T}$. If the distribution between $F_{S}(S_{s})$ and $F_{T}(S_{t})$ are similar, then we can directly apply source task Q-value predictor $V_{s}$ to the $F_{t}$, skipping the need to learn a $V_{t}$.

We first describe the domain classifier, $D$, which classifies whether encoded features are drawn from the source or the target domain. Thus $D$ is optimized according to a standard cross entropy loss, $L_{D}(S_{s},S_{t},F_{s},F_{t})$ where the labels indicate the origin domain, defined below:

And for the generator, we train with the standard loss function with inverted labels [18], then the loss can be described as:

Then, the source and target mappings are optimized according to an adversarial objective, they are optimized to confuse $D$ to unable to predict reliable domain label.

Most deep reinforcement learning algorithm can achieve great performance in single environment, but agent often cannot handle a new slightly altered environment. In Figure 2, we show the screenshot of original Pong (the leftmost column) and two variations of Pong. A DQN agent that pretrained on original Pong performs really bad on two of it’s variations (-15.9 on Pong with Gaussion noise and -20.9 on Pong with inverted color).

Rusu et al. [7] analyzed the Pong to noisy Pong case, and found that the high-level filter on the clean task is not sufficiently tolerant to the added noise. Thus some new low-level vision has to be relearned in order to adapt to the new task environment. We found that by adding some augmentations to the source task during training on the source task would help to avoid over-fitting and thus learn more key features of the task. We added a data augmentation layer before feed the input data into replay memory, the data augmentation layer will randomly transform the input. For example, Eq. (7) demonstrates a data augmentation layer that will invert the color of environment state (screen) with the probability of 30%, and remain unchanged otherwise.

While this method increases the difficulty of training the agent, but with proper augmentation setting, the difficulty of training with augmentation is almost the same as the origin (Figure 3 (a)).

In Figure 3(b), we show three different DQN agent training progress on "Pong with Gaussian noise". The “Pong with data aug.” stands for the agent trained by using the pre-trained model of Pong with augmentation defined as Eq.(7). The Baseline 1 uses random initial, and Baseline 2 uses pre-trained model of Pong. It shows that using the pre-trained model of Pong with augmentation obtains higher score within equal steps compared to other competitive methods.

The first evaluation domain is a set of synthetic variants of the Atari game “Pong”. We created synthetic variants by altering visual appearances of the original “Pong”. The variants of Pong are Noisy (Gaussian noise is added to the inputs), Grid (fixed grid lines are added on input), Invert (input color is inverted), and Scale (input is scaled by 75% and with black padding). Example frames are shown in Figure 7 (a). The purpose of this experiment is to test whether the agent can learn the core game play across it’s visual variants.

Figure 6 shows the transfer progress of each variant. Overall, our method (both with and without augmentation versions) achieve a better learning progress on target task compare to baseline methods. Our method saves at least 1 millions frames of training time to reach the convergence state for Pong-grid, Pong-scale and Pong-invert.

Moreover, we measure the transfer performance by the transfer score [7]. The transfer score is defined as the relative performance of an method compared with random baseline method. Higher transfer score means more effective transfer. Transfer score larger than 1 means the performance is relatively better than random baseline method, i.e. positive transfer. On the contrary, transfer score smaller than 1 means the performance worse than random baseline method, i.e. negative transfer.

As shown in Figure 8, we can observe that our method get better transfer scores across all experiments. Interestingly, the naive baseline (initialized with pre-trained weights on “Pong”) obtains worst transfer score. It indicates that there are negative transfer effects, especially for Pong-invert case.

Our method boosts the learning speed for both with and without augmentation versions of our method. As shown in Figure 6, we can observe that our method with augmentation learns significant faster compared to the version without augmentation. This shows that our method with augmentation can significantly help agent to learn the core game play across visual variants of a single game.

We further look closely at specified transfer pair cases. For Pong-gauss case, the difference between source and target task is smallest. We can observe that even the naïve baseline method provides a very good transfer effect. Although the naïve baseline method reaches max transfer scores (i.e. 2.0), our method still outperform it on coverage time as shown in Figure 6. For Pong-invert case, it shows that naïve baseline method fails to learn on target task, which means that the knowledge learned from source task are hindering the agent to learn target task. On the other hands, our method without augmentation minimizes the negative effect because the generator will try to produce features that are similar with source task’s features. In terms of our full method, it achieves great performance at starting time is because that the source task is trained with 30% inverted frame, in other words, the source agent has learned Pong-invert at training time, thus it can handle this specified case well. In general speaking, our method obtains the best transfer result, followed by our method without augmentation, and naïve transfer method obtains worst result.

Next, we designed a multi-level game scenario to test the effetiveness of our transfer method. It is common that for each game, the designer designed multi-levels with increasing difficuties. Instead of training the agents separately on each level, we use the proposed transfer learning method to increase the learning speed when facing harder levels.

In our scenario, we build a multi-level version of Breakout¹¹ 1 Environment source code can be found at https://github.com/SSARCandy/breakout-env with different difficulties. We designed 4 different levels with increasing difficulties by shrinking the paddle width, the different paddle widths are: 30px width (level-1), 20px width (level-2), 10px width (level-3) and 5px width (level-4). Figure 9 show 4 different levels sample frames.

And we perform 3 transfer cases, including level-1 to level-2, level-1 to level-3 and level-1 to level-4. The training progress shown as Figure 10.

We can find that the worst performance is “Random initialize” (red dashed line) because the agent was trained from scratch, without any prior knowledge. And all other transfer methods have very good transfer effectiveness and can be reached stable performance within 1 million steps. For width 30 to width 20 (level-1 to level-2) case, the performance of our methods (both w/ and w/o augmentation) are better than the Naive method, and augmentation (orange line) helps the agent to learn faster at the beginning. For width 30 to width 10 (level-1 to level-3) case, our methods still have a noticeable boost of learning effectiveness than Naive method. Finally, for with 30 to width 5 (level-1 to level-4) case, there is barely any difference between Navie method and our methods. This suggested that the difficulties of the final level (i.e. level 4) has significant different from all the previous levels. The performance of the transfer method can then be used to identify the difficulty differences during the game designing process.

Next we investigate the transfer effectiveness between different Atari games. We select 4 different games from Atari 2600 to perform cross game transfer experiments, including Pong, Breakout, SpaceInvader, and DemonAttack. We choose Pong and Breakout because they share some similarities of gameplay, i.e. the player tries to hit a moving ball with a rectangular paddle. On the other hand, SpaceInvader and DemonAttack shares similar gameplay, i.e. the player need to shoot moving enemies. In this experiments, we perform transfer between these selected games.

We report the transfer results in the transfer scores matrix shown in Figure 11. In this matrix, we compare three different methods, including naïive transfer baseline, our method w/o augmentation, and our full method.

Overall, the naïve transfer method often got transfer score that is smaller than 1, which indicates that it needs more training time compare to training on target task directly, thus have negative transfer effect. Our approaches obtain higher transfer scores than naïve transfer approach in most of the cases, which indicates that our methods help to learn the target tasks. And in some cases, although transfer scores are not high, our method eliminates the negative effect that introduced by naïve transfer method.

For our method with augmentation, the benefit of augmentation is not so significant as in Pong variants experiment described in Section 5.1. In some cases like Pong to Breakout and Pong to DemonAttack, augmentation still helps the agent learns faster. And in other cases, the performances with and without augmentation did not demonstrate noticeable differences.

We also observed that some of the transfer performances of the similar gameplay pairs (Pong and Breakout, SpaceInvader and DemonAttack) are not obviously improved. We believe that this is because these Atari games are too different than some similarity between games could not take advantage when transferring.