Actor-Critic Reinforcement Learning Method

---

---

What is Actor-Critic Method?

The actor-critic algorithm is a type of reinforcement learning that integrates policy-based techniques with value-based approaches. This combination is to overcome the limitations of employing each technique individually.

In the actor-critic framework, an agent (actor) formulates a strategy for making decisions, while a value function (critic) evaluates the actions taken by the actor. At the same time, the critic analyzes these actions by assessing their quality and value. This dual role allows the method to maintain balance between exploration and exploitation, by using the benefits of both policy and value functions.

Working of Actor-Critic Method

Actor-critic methods is a combination of policy-based and value-based techniques primarily aimed at learning a policy that enhances the expected cumulative reward. The two main components required are −

• Actor − This component is responsible for selecting actions based on the current policy. It is usually denoted as Π_θ(a|s), which represents the probability of taking action a in state s.
• Critic − The critic assesses the actions taken by the actor by estimating the value function. It is denoted by V(s), that calculates the expected return.

Step-by-step Working of Actor-Critic method

The main objective of actor-critic methods is that the actor chooses an action(policy), while the critic assesses the quality of those actions (value function), and this feedback is utilized to enhance both the actor's policy and the critic's value assessment. Following is the pseudo-algorithm for actor-critic methods −

• Begin with initializing the actor's policy parameters, critic's value function, environment and choose an initial state s₀.
• Sample {s_t,a_t} using the policy Πθ from the actor-network.
• Evaluate the advantage function. It is called as TD error δ. In actor-critic algorithm, the advantage function is generated by critic network.
• Evaluate the gradient.
• Update the policy parameters (θ)
• Adjust the critic's weights based on value-based RL. δt represents the advantage function.
• Repeat the above steps in sequence to find the optimal policy.

Advantages of Actor-Critic Method

The actor-critic method offers various advantages −

• Enhanced Sample Efficiency − The integrated approach of actor-critic algorithms results in better sample efficiency, requiring less interactions with the environment to reach optimal performance.
• Faster Convergence − The technique's capacity to simultaneously update the policy and value function leads to quicker convergence during training, allowing for quicker adaption to the learning task.
• Flexibility in Action Spaces − Actor-Critic models can effectively manage both discrete and continuous action spaces, providing adaptability for various reinforcement learning challenges.
• Off-Policy Learning − Acquires knowledge from previous experiences, even if it doesn't strictly adhere to the present policy.

Challenges of Actor-Critic Methods

Some of the key challenges of actor-critic methods that should be addressed are −

• High variance − Even with the advantage function, actor-critic methods still experience with high variance while estimating gradient. This challenge can be addressed by using methods such as Generalized Advantage Estimation (GAE).
• Training Stability − Simultaneous training of the actor and critic can lead to instability, particularly when there is a poor alignment between the actor's policy and the critic's value function. This challenge can be addressed by using techniques like Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO).
• Bias-Variance Tradeoff − The balance between bias and variance in calculating the policy gradient can occasionally result in slower convergence, making it quite challenging in the field of reinforcement learning.

Variants of Actor-Critic Methods

Some of the key variants of actor-critic method include −

• Advantage Actor-Critic (A3C) − A2C(Advantage Actor-critic) is a variant of the actor-critic algorithm that incorporates the idea of the advantage function.
• Asynchronous Advantage Actor-Critic (A3C) − This approach employs several agents operating in parallel to improve a collective policy and value function. It also helps in stabilizing training and enhancing efficiency.
• Soft Actor-Critic (SAC) − SAC is an off-policy approach that integrates entropy regularization to promote exploration. Its objective is to optimize both the expected return and the uncertainty of the policy. The key feature in SAC includes balancing exploration and exploitation by adding an entropy term to the reward.
• Deep Deterministic Policy Gradient (DDPG) − DDPG is designed for environments that involve continuous action spaces. It merges the actor-critic method with the deterministic policy gradient. The key feature of DDPG includes using a deterministic policy and a target network to stabilize training.
• Q-Prop − Q-Prop is another Actor-critic approach. In the previous methods, temporal difference learning is implemented to decrease variance allowing bias to increase. Q-Prop decreases the variance in gradient computation without introducing bias by using the idea of control variate.

Advantage Actor-Critic (A2C)

A2C(Advantage Actor-critic) is a variant of the actor-critic algorithm that incorporates the idea of the advantage function. The function evaluates the extent to which an action is better when compared to the average action in a given state. Including this advantageous information, A2C directs the learning process towards actions that are more valuable than the usual action performed in that state.

Algorithm of A2C

The steps involved in the algorithm of A2C are −

• Initialize the policy parameters, the value function parameters, and the environment.
• The agent interacts with the environment by taking actions according to the current policy and receives rewards in return.
• Calculate the Advantage Function A(s,a) based on the current policy and value estimates.
• Simultaneously update the actor's parameters using policy gradient and critic's parameters using the value-based method.

Asynchronous Advantage Actor-Critic (A3C)

The Asynchronous Advantage Actor-Critic (A3C) algorithm was introduced by Volodymyr Mnih and colleagues in 2016. This is primarily designed for employing asynchronous updates from various parallel agents, which helps in overcoming stability and sample efficiency problems found in traditional reinforcement learning algorithms.

Algorithm of A3C

The step-by-step breakdown of the A3C algorithm −

• Initialize the global network.
• Launch concurrent workers, each equipped with its individual local network. These workers interact with the environment to collect experiences (state, action, reward, next state).
• At every step throughout an episode, the worker observes the state, chooses an action based on the current policy, and receives the reward and the next state. Additionally, the worker calculates the advantage function to measure the difference between the predicted value and the actual reward that is expected.
• Update the critic (value function) and actor (policy).
• As one worker refreshes its local model, the gradients from several workers are combined asynchronously to modify the global model. This will allow the updates of each worker to be independent, which reduces correlation and leads to more stable and efficient training.

Advantage Actor-Critic (A2C) Vs. Asynchronous Advantage Actor-Critic (A3C)

The table below demonstrates the key differences between A2C (Advantage Actor-Critic) and A3C (Asynchronous Advantage Actor-Critic) −